SLURM Utilities¶
Functions for interacting with SLURM schedulers.
Overview¶
This module provides wrapper functions for common SLURM operations:
- Querying job status via
sacctandsqueue - Submitting jobs via
sbatch - Finding job output files
- Parsing SLURM output formats
Functions¶
Job Status¶
get_job_status
¶
Get the state of one or more jobs.
Convenience wrapper around get_sacct_info that returns only job states.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
job_ids
|
Union[str, List[str]]
|
Single job ID or list of job IDs. |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, str]
|
Dictionary mapping JobID to State string. |
Example
get_job_status(["12345", "12346"])
Source code in src/slurmkit/slurm.py
get_sacct_info
¶
get_sacct_info(job_ids: Union[str, List[str]], fields: Optional[List[str]] = None) -> Dict[str, Dict[str, str]]
Query sacct for job metadata.
Queries the SLURM accounting database for information about completed or running jobs. Results are filtered to exclude job steps (e.g., .batch, .extern) and only return the main job entry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
job_ids
|
Union[str, List[str]]
|
Single job ID or list of job IDs to query. |
required |
fields
|
Optional[List[str]]
|
List of sacct fields to retrieve. Defaults to DEFAULT_SACCT_FIELDS.
Common fields: State, Elapsed, Start, End, ExitCode, AllocCPUS,
ReqMem, NodeList, Partition. Run |
None
|
Returns:
| Type | Description |
|---|---|
Dict[str, Dict[str, str]]
|
Dictionary mapping JobID to a dict of field values. |
Dict[str, Dict[str, str]]
|
Returns empty dict if sacct is not available or no jobs found. |
Example
info = get_sacct_info(["12345", "12346"]) info["12345"]["State"] 'COMPLETED' info["12345"]["Elapsed"] '01:23:45'
Source code in src/slurmkit/slurm.py
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | |
get_canonical_sacct_states
¶
Query sacct full rows and resolve canonical state per parent SLURM job ID.
Canonical resolution order:
1) Any RUNNING/COMPLETING row => RUNNING
2) Else any PENDING/REQUEUED/SUSPENDED row => PENDING
3) Else terminal mapping using .batch first, then parent fallback
Source code in src/slurmkit/slurm.py
get_pending_jobs
¶
Query squeue for currently pending or running jobs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
user
|
Optional[str]
|
Username to filter jobs. If None, uses current user (--me). |
None
|
Returns:
| Type | Description |
|---|---|
List[Dict[str, str]]
|
List of dicts with keys: 'job_id', 'job_name', 'state', 'wait_time'. |
List[Dict[str, str]]
|
wait_time is formatted as human-readable string for pending jobs. |
Example
pending = get_pending_jobs() pending[0]
Source code in src/slurmkit/slurm.py
get_sacct_info() preserves the legacy parent-row-only view (job_id without
step rows). get_canonical_sacct_states() evaluates full sacct rows
(job_id, .batch, .extern, other steps), returns canonical state, and
includes row-level raw_state diagnostics.
Job Submission¶
submit_job
¶
submit_job(script_path: Union[str, Path], dry_run: bool = False, extra_args: Optional[List[str]] = None) -> Tuple[bool, Optional[str], str]
Submit a job script using sbatch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
script_path
|
Union[str, Path]
|
Path to the job script. |
required |
dry_run
|
bool
|
If True, print command without executing. |
False
|
extra_args
|
Optional[List[str]]
|
Additional arguments to pass to sbatch. |
None
|
Returns:
| Type | Description |
|---|---|
bool
|
Tuple of (success, job_id, message). |
Optional[str]
|
|
str
|
|
Tuple[bool, Optional[str], str]
|
|
Example
success, job_id, msg = submit_job("jobs/train.job") print(f"Submitted job {job_id}") Submitted job 12345
Source code in src/slurmkit/slurm.py
Job Output¶
find_job_output
¶
find_job_output(job_id: str, jobs_dir: Optional[Path] = None, config: Optional[Config] = None) -> List[Path]
Find job output files matching a job ID.
Searches for .out files containing the job ID in their filename, using configured output patterns to match files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
job_id
|
str
|
SLURM job ID to search for. |
required |
jobs_dir
|
Optional[Path]
|
Directory to search in. If None, uses config's jobs_dir. |
None
|
config
|
Optional[Config]
|
Configuration object. If None, uses global config. |
None
|
Returns:
| Type | Description |
|---|---|
List[Path]
|
List of matching file paths, sorted by modification time (newest first). |
Example
outputs = find_job_output("12345") outputs[0] PosixPath('jobs/exp1/logs/train_model.12345.out')
Source code in src/slurmkit/slurm.py
get_job_script_path
¶
Get the script path for a job using scontrol.
This only works for jobs that are still in SLURM's active job table (typically running or recently completed jobs).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
job_id
|
str
|
SLURM job ID. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Path]
|
Path to the job script, or None if not found. |
Source code in src/slurmkit/slurm.py
Usage Example¶
from slurmkit import (
get_job_status,
get_sacct_info,
get_pending_jobs,
submit_job,
find_job_output,
)
# Get status of specific jobs
statuses = get_job_status(["12345", "12346", "12347"])
for job_id, status in statuses.items():
print(f"Job {job_id}: {status}")
# Get detailed accounting info
info = get_sacct_info(["12345"])
print(info) # DataFrame with job details
# List pending/running jobs
pending = get_pending_jobs()
print(f"You have {len(pending)} jobs in the queue")
# Submit a job script
job_id = submit_job("jobs/exp1/job_scripts/train.sh")
print(f"Submitted job: {job_id}")
# Find output file for a job
output_path = find_job_output(
job_id="12345",
job_name="lr0.01_bs32",
search_dirs=["jobs/exp1/logs"]
)
print(f"Output file: {output_path}")