Skip to content

Notifications

slurmkit supports webhook/chat and SMTP email notifications for job and collection lifecycle events.

Supported transport types: - webhook - slack - discord - email (SMTP)

Default behavior remains failure-first: - notify job defaults to --on failed - route event subscription defaults to job_failed

Configuration

Add notification settings to .slurmkit/config.yaml:

notifications:
  defaults:
    events: [job_failed]
    timeout_seconds: 5
    max_attempts: 3
    backoff_seconds: 0.5
    output_tail_lines: 40

  formatter:
    callback: null  # "module.path:function_name"

  job:
    ai:
      enabled: false
      callback: null

  collection_final:
    attempt_mode: latest
    min_support: 3
    top_k: 10
    include_failed_output_tail_lines: 20
    ai:
      enabled: false
      callback: null

  routes:
    - name: team_slack
      type: slack
      url: "${SLACK_WEBHOOK_URL}"
      events: [job_failed, collection_failed]
      formatter_callback: null

    - name: personal_discord
      type: discord
      url: "${DISCORD_WEBHOOK_URL}"
      events: [job_failed, job_completed]

    - name: custom_ops
      type: webhook
      url: "${OPS_WEBHOOK_URL}"
      headers:
        Authorization: "Bearer ${OPS_WEBHOOK_TOKEN}"
      events: [job_failed, collection_failed, collection_completed]

    - name: team_email
      type: email
      to: ["ops@example.com", "ml@example.com"]
      from: "${SLURMKIT_EMAIL_FROM}"
      smtp_host: "${SMTP_HOST}"
      smtp_port: 587
      smtp_username: "${SMTP_USER}"
      smtp_password: "${SMTP_PASSWORD}"
      smtp_starttls: true
      smtp_ssl: false
      events: [job_failed, collection_failed]

Route-level settings override notifications.defaults.

Formatter callback precedence: 1. notifications.routes[].formatter_callback (if provided) 2. notifications.formatter.callback (global) 3. Built-in default formatter

Set formatter_callback: null (or empty string) on a route to opt out of global callback.

url, headers, and email fields support ${ENV_VAR} interpolation. Missing env vars are reported as route configuration errors at runtime.

For type: email: - required fields: to, from, smtp_host - to supports string, comma-separated string, or list of strings - smtp_starttls and smtp_ssl cannot both be true - smtp_username and smtp_password must be set together (or both omitted)

Formatter callback contract (module:function): - Signature: def format_notification(payload: Dict[str, Any]) -> Dict[str, Any] - Payload includes meta.route_name and meta.route_type. - Supported return keys: - chat for slack/discord - email_subject for email - email_body for email - Invalid callback output is partially ignored by field and missing/invalid fields fall back to built-in formatter output.

Collection-Specific Overrides from spec.yaml

When a collection was created via slurmkit generate ..., slurmkit stores generation.spec_path in the collection metadata. notify job and notify collection-final use this precedence:

  1. Top-level notifications in the collection's spec.yaml
  2. Global .slurmkit/config.yaml notifications

Merge semantics: - Dictionary fields deep-merge (spec values override global values). - List fields replace (not append), including notifications.routes.

If the spec file is missing, unreadable, or malformed, slurmkit prints a context warning and falls back to the global config.

Example job_spec.yaml override:

notifications:
  defaults:
    output_tail_lines: 15
  job:
    ai:
      enabled: true
      callback: "utilities.slurmkit.ai_callbacks:summarize_job_payload"
  collection_final:
    ai:
      enabled: true
      callback: "utilities.slurmkit.ai_callbacks:summarize_collection_report"

See the runnable demo in examples/demo_project/README.md.

Commands

Test routes

slurmkit notify test
slurmkit notify test --route team_slack
slurmkit notify test --dry-run
slurmkit notify test --route team_email --dry-run
# then run live (no --dry-run) once SMTP/env vars are configured
slurmkit notify test --route team_email

Send job notification

slurmkit notify job --job-id 12345 --exit-code 1
slurmkit notify job --job-id 12345 --exit-code 0 --on always
slurmkit notify job --job-id 12345 --route team_slack --route custom_ops
slurmkit notify job --job-id 12345 --dry-run

Send collection-final notification

slurmkit notify collection-final --job-id 12345
slurmkit notify collection-final --job-id 12345 --collection exp1
slurmkit notify collection-final --job-id 12345 --trigger-exit-code 0
slurmkit notify collection-final --job-id 12345 --route custom_ops --strict
slurmkit notify collection-final --job-id 12345 --no-refresh --dry-run

notify collection-final behavior: - refreshes collection state before finality check (unless --no-refresh) - uses latest-attempt semantics to classify each logical job - sends collection_completed or collection_failed only when collection is terminal - can treat collection as terminal when the only active effective row is the trigger job (--job-id) - when that fallback path is used: --trigger-exit-code 0 infers completed, non-zero infers failed - if fallback is used without --trigger-exit-code, trigger state is inferred as unknown and a warning is printed - deduplicates by terminal-state fingerprint (use --force to bypass)

Job Script Integration

Recommended shell pattern to preserve original job exit code while triggering both job and collection-final notifications:

trap 'rc=$?; slurmkit notify job --job-id "${SLURM_JOB_ID}" --exit-code "${rc}"; slurmkit notify collection-final --job-id "${SLURM_JOB_ID}" --trigger-exit-code "${rc}"; exit "${rc}"' EXIT

If your script handles exits explicitly, use equivalent end-of-script flow:

rc=$?
slurmkit notify job --job-id "${SLURM_JOB_ID}" --exit-code "${rc}"
slurmkit notify collection-final --job-id "${SLURM_JOB_ID}" --trigger-exit-code "${rc}"
exit "${rc}"

Event Model

notify job derives event from exit code: - exit_code != 0 -> job_failed - exit_code == 0 -> job_completed

notify collection-final emits: - collection_failed when terminal and any effective job is failed/unknown - collection_completed when terminal and all effective jobs are completed

By default (--on failed), completed job notifications are skipped.

Canonical Webhook Payload

For type: webhook, slurmkit sends stable schema_version: v1 JSON.

Job payload includes: - event - generated_at - context_source (collection_match, env_only, ambiguous_match) - job - collection (when uniquely resolved) - ai_status - optional ai_summary - host - meta (route name/type)

Collection-final payload additionally includes: - collection_report (summary counts, failed rows, risky/stable values, recommendations) - trigger_job_id - ai_status - optional ai_summary

Test payload includes deterministic baseline fields and does not run AI callbacks.

For slack/discord, slurmkit sends a human-readable summary with key metadata. Optional formatter callbacks can override message text via chat.

For email, slurmkit sends plain-text emails through SMTP with built-in subject/body formatting. Optional formatter callbacks can override email_subject and email_body.

If formatter callbacks fail (load/execute/validation), delivery still proceeds with built-in formatter output and warnings are shown in CLI output.