Serialize hash-pinned dynamo install per node (fix concurrent pip race)

[Ankur-singh]

Problem

srtctl's dynamo.install: true runs its pip installs (and, for top-of-tree, the whole clone+build) on every task/rank. With pyxis/enroot, all tasks on a node that share a --container-image share one container instance — one rootfs, one site-packages, one /tmp. So N concurrent installs race on those shared paths.

On containers that pre-ship ai-dynamo's Python deps at conflicting versions (e.g. nvcr.io/nvidia/tensorrt-llm/release, which carries pydantic / pydantic-core / typing-extensions / uvloop for TRT-LLM), --force-reinstall tears those down concurrently and dies:

Uninstalling typing_extensions-4.15.0:
ERROR: Could not install packages due to an OSError:
       [Errno 2] No such file or directory: '.../__pycache__/typing_extensions.cpython-312.pyc'
*** STEP … CANCELLED … DUE TO TASK FAILURE ***

Repro: a Qwen3.5 dynamo-trt disagg sweep on the TRT-LLM rc20 release container (TP2 prefill / TP4 decode → multiple tasks per node) — every worker log shows the doubled Processing …/ai_dynamo_runtime.whl and the OSError above (NVIDIA/InferenceMAX#133).

SGLang dynamo.hash configs run the same code path but don't fail: their container doesn't pre-ship those conflicting deps, so --force-reinstall isn't removing shared files out from under a racing sibling.

Fix

Serialize the install per node with a node-local flock + sentinel in /tmp (site-packages is per node / per container instance): the first local task installs, the rest block on the lock then skip. This is the desired end state anyway — one shared install per node that all ranks import — just without concurrent writers.

Both install paths are guarded:

• _hash_cached_source_install (dynamo.hash) — the shared /configs build-flock is unchanged; the new node-local guard wraps only the post-build pip installs. Sentinel keyed by hash.
• _live_source_install_for_top_of_tree (dynamo.top_of_tree) — had no flock at all, and both the SGLang and portable branches clone/build/install into shared paths (/sgl-workspace or /tmp, plus site-packages). The whole install is now wrapped in the same node-local guard. Fixed sentinel key (HEAD has no hash; a container instance only ever installs one top-of-tree build).

Successful run: https://github.com/NVIDIA/InferenceMAX/actions/runs/28761452215/job/85277686958?pr=133

Test

Extended test_hash_install_command and test_top_of_tree_install_command to assert the node-local guard (flock -x 201, the sentinel check/touch, the 201> lock file); existing substring assertions still hold. tests/test_configs.py is fully green (112 passed). Full suite passes except two pre-existing, unrelated console-rendering assertions (test_apply_json::test_apply_without_json_emits_prose_only, test_dry_run::…test_srun_options_shown) that also fail on clean main in a narrow terminal.