Problem
The cluster-wide default_bash_preamble setting (introduced in #104) is only applied to srun calls that go through the use_bash_wrapper=True path in start_srun_process (src/srtctl/core/slurm.py). Call sites that pre-wrap their command in bash -c "..." and pass use_bash_wrapper=False skip the preamble entirely — a warning is logged but the preamble is not applied.
The most visible casualty is nginx (src/srtctl/cli/mixins/frontend_stage.py:_start_nginx), which is launched with use_bash_wrapper=False and therefore does not pick up cluster-wide ulimit -n settings. We also work around this in the nginx template via worker_rlimit_nofile, plus an inlined ulimit -n in the nginx bash command.
Relevant code:
# src/srtctl/core/slurm.py
else:
cluster_preamble = _get_cluster_bash_preamble()
if cluster_preamble:
logger.warning(
"Cluster default_bash_preamble is set but this srun bypasses the bash wrapper "
"(use_bash_wrapper=False); preamble will not be applied. command=%s",
shlex.join(command),
)
srun_cmd.extend(command)
Proposed fix
Make start_srun_process apply the cluster preamble unconditionally — when use_bash_wrapper=False and a cluster preamble is configured, wrap the command in bash -c "<preamble> && <orig command>" instead of skipping it. This way default_bash_preamble is truly cluster-wide and individual call sites don't need to inline workarounds.
Cleanup once landed
- Remove the inlined
ulimit -n 1048576 && ... from _start_nginx in src/srtctl/cli/mixins/frontend_stage.py.
- Decide whether
worker_rlimit_nofile in nginx.conf.j2 stays (probably yes — it's nginx-idiomatic and independent of the shell limit).
Problem
The cluster-wide
default_bash_preamblesetting (introduced in #104) is only applied to srun calls that go through theuse_bash_wrapper=Truepath instart_srun_process(src/srtctl/core/slurm.py). Call sites that pre-wrap their command inbash -c "..."and passuse_bash_wrapper=Falseskip the preamble entirely — a warning is logged but the preamble is not applied.The most visible casualty is nginx (
src/srtctl/cli/mixins/frontend_stage.py:_start_nginx), which is launched withuse_bash_wrapper=Falseand therefore does not pick up cluster-wideulimit -nsettings. We also work around this in the nginx template viaworker_rlimit_nofile, plus an inlinedulimit -nin the nginx bash command.Relevant code:
Proposed fix
Make
start_srun_processapply the cluster preamble unconditionally — whenuse_bash_wrapper=Falseand a cluster preamble is configured, wrap the command inbash -c "<preamble> && <orig command>"instead of skipping it. This waydefault_bash_preambleis truly cluster-wide and individual call sites don't need to inline workarounds.Cleanup once landed
ulimit -n 1048576 && ...from_start_nginxinsrc/srtctl/cli/mixins/frontend_stage.py.worker_rlimit_nofileinnginx.conf.j2stays (probably yes — it's nginx-idiomatic and independent of the shell limit).