Job Generation¶
Template-based SLURM job script generation with parameter sweeps.
Overview¶
The JobGenerator class generates SLURM job scripts from Jinja2 templates combined with parameter specifications. It supports:
- Grid-based parameter sweeps
- Explicit parameter lists
- Dynamic SLURM argument computation via Python functions
- Custom job naming patterns
Classes¶
JobGenerator¶
JobGenerator
¶
JobGenerator(template_path: Union[str, Path], parameters: Dict[str, Any], slurm_defaults: Optional[Dict[str, Any]] = None, slurm_logic_file: Optional[Union[str, Path]] = None, slurm_logic_function: str = 'get_slurm_args', job_name_pattern: Optional[str] = None, logs_dir: Optional[Union[str, Path]] = None, config: Optional[Config] = None)
Generator for SLURM job scripts from templates and parameters.
The generator combines: - Jinja2 templates for job script structure - Parameter grids or lists for job variations - Optional Python logic for SLURM argument customization - Collection integration for tracking generated jobs
Example
generator = JobGenerator( ... template_path="templates/train.job.j2", ... parameters={"mode": "grid", "values": {"lr": [0.01, 0.1]}}, ... slurm_defaults={"partition": "gpu", "time": "24:00:00"}, ... ) scripts = generator.generate(output_dir="jobs/exp1/job_scripts")
Initialize the job generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
template_path
|
Union[str, Path]
|
Path to the Jinja2 template file. |
required |
parameters
|
Dict[str, Any]
|
Parameter specification (mode + values). |
required |
slurm_defaults
|
Optional[Dict[str, Any]]
|
Default SLURM arguments. |
None
|
slurm_logic_file
|
Optional[Union[str, Path]]
|
Path to Python file with SLURM args function. |
None
|
slurm_logic_function
|
str
|
Name of the function in slurm_logic_file. |
'get_slurm_args'
|
job_name_pattern
|
Optional[str]
|
Jinja2 pattern for job names. |
None
|
logs_dir
|
Optional[Union[str, Path]]
|
Directory for job output files (used in templates). |
None
|
config
|
Optional[Config]
|
Configuration object. |
None
|
Source code in src/slurmkit/generate.py
722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 | |
from_spec
classmethod
¶
from_spec(spec_path: Union[str, Path], config: Optional[Config] = None, collection_name: Optional[str] = None, interpolation_context: Optional[Mapping[str, Any]] = None) -> 'JobGenerator'
Create a generator from a job spec file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec_path
|
Union[str, Path]
|
Path to the job spec YAML file. |
required |
config
|
Optional[Config]
|
Configuration object. |
None
|
collection_name
|
Optional[str]
|
Optional collection name used in path interpolation. |
None
|
interpolation_context
|
Optional[Mapping[str, Any]]
|
Optional explicit interpolation context. |
None
|
Returns:
| Type | Description |
|---|---|
'JobGenerator'
|
Configured JobGenerator instance. |
Source code in src/slurmkit/generate.py
generate
¶
generate(output_dir: Union[str, Path], collection: Optional[Collection] = None, dry_run: bool = False) -> List[Dict[str, Any]]
Generate job scripts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_dir
|
Union[str, Path]
|
Directory to write generated scripts. |
required |
collection
|
Optional[Collection]
|
Collection to add generated jobs to. |
None
|
dry_run
|
bool
|
If True, don't write files, just return what would be generated. |
False
|
Returns:
| Type | Description |
|---|---|
List[Dict[str, Any]]
|
List of generated job info dicts with keys: |
List[Dict[str, Any]]
|
|
List[Dict[str, Any]]
|
|
List[Dict[str, Any]]
|
|
List[Dict[str, Any]]
|
|
Source code in src/slurmkit/generate.py
Functions¶
expand_grid
¶
expand_grid(params: Dict[str, List[Any]], filter_func: Optional[Callable[[Dict[str, Any]], bool]] = None) -> Iterator[Dict[str, Any]]
Expand a parameter grid into all combinations.
Takes a dictionary where values are lists and yields all combinations as individual dictionaries.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
params
|
Dict[str, List[Any]]
|
Dictionary mapping parameter names to lists of values. |
required |
filter_func
|
Optional[Callable[[Dict[str, Any]], bool]]
|
Optional predicate to include/exclude combinations. If provided, only combinations where filter_func(params) is True are yielded. |
None
|
Yields:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dictionary for each combination of parameter values. |
Example
list(expand_grid({"a": [1, 2], "b": ["x", "y"]})) [{'a': 1, 'b': 'x'}, {'a': 1, 'b': 'y'}, {'a': 2, 'b': 'x'}, {'a': 2, 'b': 'y'}]
Source code in src/slurmkit/generate.py
expand_parameters
¶
expand_parameters(spec: Dict[str, Any], filter_func: Optional[Callable[[Dict[str, Any]], bool]] = None, parse_func: Optional[Callable[[Dict[str, Any]], Union[Dict[str, Any], List[Dict[str, Any]]]]] = None) -> List[Dict[str, Any]]
Expand a parameter specification into a list of parameter dicts.
Supports two modes: - "grid": All combinations of parameter values - "list": Explicit list of parameter dictionaries
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
Dict[str, Any]
|
Parameter specification with "mode" and "values" keys. mode: "grid" or "list" values: For grid mode, dict of param_name -> list of values. For list mode, list of param dicts. |
required |
filter_func
|
Optional[Callable[[Dict[str, Any]], bool]]
|
Optional predicate for grid mode to include/exclude combinations. |
None
|
parse_func
|
Optional[Callable[[Dict[str, Any]], Union[Dict[str, Any], List[Dict[str, Any]]]]]
|
Optional callback to derive effective parameters for each generated job before filtering and rendering. The callback may return a single dict or a list of dicts. |
None
|
Returns:
| Type | Description |
|---|---|
List[Dict[str, Any]]
|
List of parameter dictionaries. |
Example
spec = {"mode": "grid", "values": {"lr": [0.01, 0.1], "bs": [32]}} expand_parameters(spec) [{'lr': 0.01, 'bs': 32}, {'lr': 0.1, 'bs': 32}]
spec = {"mode": "list", "values": [{"lr": 0.01}, {"lr": 0.1}]} expand_parameters(spec) [{'lr': 0.01}, {'lr': 0.1}]
Source code in src/slurmkit/generate.py
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
load_job_spec
¶
Load a job specification from a YAML file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec_path
|
Union[str, Path]
|
Path to the job spec YAML file. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Parsed job specification dictionary. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If the file doesn't exist. |
Source code in src/slurmkit/generate.py
generate_job_name
¶
generate_job_name(params: Dict[str, Any], pattern: Optional[str] = None, env: Optional[Environment] = None) -> str
Generate a job name from parameters.
Uses a Jinja2 pattern to format the job name from parameter values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
params
|
Dict[str, Any]
|
Job parameters. |
required |
pattern
|
Optional[str]
|
Jinja2 pattern for job name. If None, uses param values joined by underscores. |
None
|
env
|
Optional[Environment]
|
Jinja2 environment. If None, creates a new one. |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Generated job name. |
Example
generate_job_name({"model": "resnet", "lr": 0.01}, "{{ model }}_lr{{ lr }}") 'resnet_lr0.01'
Source code in src/slurmkit/generate.py
Usage Example¶
from slurmkit import JobGenerator
# Create generator from a job spec file
generator = JobGenerator.from_spec("experiments/exp1/job_spec.yaml")
# Generate all job scripts
jobs = generator.generate()
for job in jobs:
print(f"Generated: {job['script_path']}")
print(f" Parameters: {job['params']}")
print(f" Job name: {job['job_name']}")
Manual Configuration¶
from slurmkit.generate import JobGenerator, expand_grid
# Expand parameter grid
params_list = expand_grid({
"learning_rate": [0.001, 0.01, 0.1],
"batch_size": [32, 64]
})
# Create generator manually
generator = JobGenerator(
template_path="templates/train.job.j2",
output_dir="jobs/exp1/job_scripts",
parameters=params_list,
slurm_defaults={"partition": "gpu", "time": "24:00:00"},
job_name_pattern="lr{{ learning_rate }}_bs{{ batch_size }}"
)
# Generate scripts
jobs = generator.generate()