Project: HPC Submission Launcher for Hydra
Author: Carel van Niekerk
Year: 2024
Group: Dialogue Systems and Machine Learning Group
Institution: Heinrich Heine University Düsseldorf
uv add git+https://github.com/carelvniekerk/Hydra-HPC-Launcher.gitAdd to your Hydra Config:
defaults:
- override hydra/launcher: hpc_submissionOr to your hydra zen wrapper:
@store(
name="main",
hydra_defaults=["_self_", {"override hydra/launcher": "hpc_submission"}],
)Note that the default hydra launcher and sweeper are called basic.
For each script that you want to use the launcher for, you need to register the plugin.
A good place to do this is in your package's __init__.py file, so that the plugin is registered whenever you import your package.
Note: The plugin_name you pass to register_plugin() must match the name used in your Hydra config (e.g., hpc_submission in the example above).
try:
print("Registering hpc_submission_launcher hydra plugin ...")
from hydra_hpc_launcher import HPCSubmissionConfig, HPCSubmissionLauncher, register_plugin
from hydra_hpc_launcher.launcher.types import Template
hpc_configuration = HPCSubmissionConfig(
template=Template.A100_40GB,
walltime="48:00:00",
)
register_plugin(
plugin_name="hpc_submission",
config=hpc_configuration,
launcher=HPCSubmissionLauncher,
)
print("hpc_submission_launcher hydra plugin registered.")
except ImportError as e:
print("WARNING: hpc_submission_launcher hydra plugin not found!")
print(f"ImportError:\n{e}")
print("If you do not intend to use Hydra HPC submission features, you can safely ignore this warning.")
except AttributeError as e:
print("WARNING: hpc_submission_launcher hydra plugin registration failed!")
print(f"AttributeError:\n{e}")
print("If you do not intend to use Hydra HPC submission features, you can safely ignore this warning.")