Skip to content

default_bash_preamble silently skipped when use_bash_wrapper=False (e.g. nginx) #107

Description

@ishandhanani

Problem

The cluster-wide default_bash_preamble setting (introduced in #104) is only applied to srun calls that go through the use_bash_wrapper=True path in start_srun_process (src/srtctl/core/slurm.py). Call sites that pre-wrap their command in bash -c "..." and pass use_bash_wrapper=False skip the preamble entirely — a warning is logged but the preamble is not applied.

The most visible casualty is nginx (src/srtctl/cli/mixins/frontend_stage.py:_start_nginx), which is launched with use_bash_wrapper=False and therefore does not pick up cluster-wide ulimit -n settings. We also work around this in the nginx template via worker_rlimit_nofile, plus an inlined ulimit -n in the nginx bash command.

Relevant code:

# src/srtctl/core/slurm.py
else:
    cluster_preamble = _get_cluster_bash_preamble()
    if cluster_preamble:
        logger.warning(
            "Cluster default_bash_preamble is set but this srun bypasses the bash wrapper "
            "(use_bash_wrapper=False); preamble will not be applied. command=%s",
            shlex.join(command),
        )
    srun_cmd.extend(command)

Proposed fix

Make start_srun_process apply the cluster preamble unconditionally — when use_bash_wrapper=False and a cluster preamble is configured, wrap the command in bash -c "<preamble> && <orig command>" instead of skipping it. This way default_bash_preamble is truly cluster-wide and individual call sites don't need to inline workarounds.

Cleanup once landed

  • Remove the inlined ulimit -n 1048576 && ... from _start_nginx in src/srtctl/cli/mixins/frontend_stage.py.
  • Decide whether worker_rlimit_nofile in nginx.conf.j2 stays (probably yes — it's nginx-idiomatic and independent of the shell limit).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions