Every job on the cluster runs through Slurm. Each Spark has one GB10 GPU, so a job asks for at most one GPU per node.
A first GPU job
#!/bin/bash
#SBATCH --job-name=gpu-check
#SBATCH --partition=jobs
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=01:00:00
srun --container-image=nvcr.io#nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi -LSubmit it from Open OnDemand’s Job Composer. The output file should list one GB10 GPU.
The first run takes longer, about 80 seconds, because Slurm imports the container image before it starts. That run also pauses the shared models for everyone for about 3 minutes, as below explains. Chat and the API may say the models are paused; they come back on their own.
Ask for memory
Each Spark has 128 GB in total, and the system keeps some of it. Asking for up to about 100 GB on one node is safe.
Containers
Jobs run containers through NVIDIA’s Pyxis and Enroot. Add --container-image to srun:
--container-image=ubuntu:24.04pulls from Docker Hub.--container-image=nvcr.io#nvidia/pytorch:<tag>pulls from NVIDIA’s NGC catalog. Note the#after the registry name.
The Sparks use Arm (aarch64) CPUs, so pick images built for Arm. NVIDIA’s NGC images for CUDA and PyTorch include Arm builds.
Limits
| Partition | jobs (the default) |
| Longest run | 7 days (--time=7-00:00:00) |
| GPUs | 1 per node (--gres=gpu:1) |
| CPUs | up to 18 or 20 per node |
Jobs pause the shared models
The models behind chat and the API run as a Slurm job of their own, in an admin-only serve partition, on both Sparks. Your jobs come first: when a job in jobs needs those nodes, Slurm stops the models and puts them back in the queue. While the cluster has two Sparks, that’s almost any job.
While the models are paused, chat answers with a message saying they’re paused and come back on their own, and the API returns 503 with Retry-After: 60. Nothing is lost: wait a few minutes and try again. The OpenAI SDKs retry the 503 by themselves; in your own code, retry with a delay rather than in a tight loop.
To keep the pause short:
- Ask for the time you need with
--time, not the 7-day maximum. Slurm ends the job at that limit, so the models come back on time. - Test with small, short runs before a long one.
- Cancel jobs you no longer want with
scancel <jobid>. - Run long jobs when the cluster is quiet.
#SBATCH --begin=22:00holds a job until 10 PM, and--begin=now+2hoursworks too.
Check what’s running
Open OnDemand’s Active Jobs shows the whole queue. From a shell, run:
squeueA tensorfold-serve job in the serve partition marked R means the models are running. PD means it’s waiting for jobs to finish. Your own jobs, and how long they ran:
squeue --me
sacct -X --format=JobID,JobName,Partition,State,Start,End,ElapsedOnce the four new Sparks arrive, the models will move to nodes of their own and jobs will stop pausing them.