Getting Started¶
This guide walks through the current slurmkit workflow: initialize config, generate a collection, submit it, monitor it, and retry failures.
Install¶
From source¶
From PyPI¶
Initialize a project¶
From your project root:
This creates .slurmkit/config.yaml and the derived metadata directories:
.slurmkit/collections/.slurmkit/sync/.slurmkit/locks/collections/
By default, generated job artifacts go under .jobs/.
Minimal config¶
jobs_dir: .jobs/
output_patterns:
- "{job_name}.{job_id}.out"
- "{job_name}.{job_id}.*.out"
- "slurm-{job_id}.out"
slurm_defaults:
partition: compute
time: "24:00:00"
mem: "16G"
ui:
mode: auto
interactive: true
show_banner: true
columns:
collections_show:
- job_name
- job_id
- state
- runtime
- eta_completion
- attempt
- submission_group
- resubmissions
- output_path
collections_show:
pager: less # less | none
collections_dir, sync_dir, and job subdirectory names are no longer user-configurable. They are fixed under .slurmkit/ and .jobs/.
Create a job spec¶
name: my_experiment
description: "Hyperparameter sweep"
template: train.job.j2
job_subdir: experiments/my_experiment
parameters:
mode: grid
values:
learning_rate: [0.001, 0.01]
batch_size: [32, 64]
slurm_args:
defaults:
partition: gpu
time: "08:00:00"
mem: "32G"
job_name_pattern: "lr{{ learning_rate }}_bs{{ batch_size }}"
With this spec and the default config, slurmkit writes:
- scripts to
.jobs/experiments/my_experiment/job_scripts/ - logs to
.jobs/experiments/my_experiment/logs/
Generate a collection¶
Interactive mode helps when arguments are missing:
slurmkitslurmkit generateslurmkit submitslurmkit status
In non-interactive mode or with --nointeractive, required args must be provided explicitly.
Submit and inspect¶
slurmkit submit my_experiment --dry-run
slurmkit submit my_experiment
slurmkit status my_experiment
slurmkit collections show my_experiment
slurmkit collections analyze my_experiment
Refresh collection state from SLURM explicitly:
Refresh computes a canonical attempt state from full sacct rows (including
.batch / .extern) and stores row-level diagnostics in each attempt's
raw_state field.
Retry failures¶
slurmkit resubmit my_experiment --filter failed --dry-run
slurmkit resubmit my_experiment --filter failed
Collection resubmission regenerates scripts by default and stores retry attempts in the collection history.
Notifications¶
slurmkit notify test --dry-run
slurmkit notify job --job-id 123456 --exit-code 1 --dry-run
slurmkit notify collection-final --collection my_experiment --job-id 123456 --dry-run
Migrate old projects¶
If a repo still uses .slurm-kit/, .job-collections/, or specs with output_dir / logs_dir, run:
Migration rewrites local state into the current .slurmkit/ and job_subdir model and stores backups under .slurmkit/backups/.
Demo project¶
The repo includes a runnable demo in examples/demo_project: