Skip to content

Slurm jobs

Write a Slurm job that asks for a GPU and runs in a container on a DGX Spark.

Every job on the cluster runs through Slurm. Each Spark has one GB10 GPU, so a job asks for at most one GPU per node.

A first GPU job

#!/bin/bash
#SBATCH --job-name=gpu-check
#SBATCH --partition=jobs
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=01:00:00

srun --container-image=nvcr.io#nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi -L

Submit it from Open OnDemand’s Job Composer. The output file should list one GB10 GPU.

The first run takes longer, about 80 seconds, because Slurm imports the container image before it starts. That run also pauses the shared models for everyone for about 3 minutes, as below explains. Chat and the API may say the models are paused; they come back on their own.

Ask for memory

Each Spark has 128 GB in total, and the system keeps some of it. Asking for up to about 100 GB on one node is safe.

Containers

Jobs run containers through NVIDIA’s Pyxis and Enroot. Add --container-image to srun:

  • --container-image=ubuntu:24.04 pulls from Docker Hub.
  • --container-image=nvcr.io#nvidia/pytorch:<tag> pulls from NVIDIA’s NGC catalog. Note the # after the registry name.

The Sparks use Arm (aarch64) CPUs, so pick images built for Arm. NVIDIA’s NGC images for CUDA and PyTorch include Arm builds.

Limits

Partition jobs (the default)
Longest run 7 days (--time=7-00:00:00)
GPUs 1 per node (--gres=gpu:1)
CPUs up to 18 or 20 per node

Jobs pause the shared models

The models behind chat and the API run as a Slurm job of their own, in an admin-only serve partition, on both Sparks. Your jobs come first: when a job in jobs needs those nodes, Slurm stops the models and puts them back in the queue. While the cluster has two Sparks, that’s almost any job.

While the models are paused, chat answers with a message saying they’re paused and come back on their own, and the API returns 503 with Retry-After: 60. Nothing is lost: wait a few minutes and try again. The OpenAI SDKs retry the 503 by themselves; in your own code, retry with a delay rather than in a tight loop.

To keep the pause short:

  • Ask for the time you need with --time, not the 7-day maximum. Slurm ends the job at that limit, so the models come back on time.
  • Test with small, short runs before a long one.
  • Cancel jobs you no longer want with scancel <jobid>.
  • Run long jobs when the cluster is quiet. #SBATCH --begin=22:00 holds a job until 10 PM, and --begin=now+2hours works too.

Check what’s running

Open OnDemand’s Active Jobs shows the whole queue. From a shell, run:

squeue

A tensorfold-serve job in the serve partition marked R means the models are running. PD means it’s waiting for jobs to finish. Your own jobs, and how long they ran:

squeue --me
sacct -X --format=JobID,JobName,Partition,State,Start,End,Elapsed

Once the four new Sparks arrive, the models will move to nodes of their own and jobs will stop pausing them.

Type to search…

↑↓ navigate↵ selectEsc close