---
title: "Slurm jobs"
description: "Write a Slurm job that asks for a GPU and runs in a container on a DGX Spark."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.c4e.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Slurm jobs

Every job on the cluster runs through [Slurm](https://slurm.schedmd.com/documentation.html). Each Spark has one GB10 GPU, so a job asks for at most one GPU per node.

## A first GPU job

```bash
#!/bin/bash
#SBATCH --job-name=gpu-check
#SBATCH --partition=jobs
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=01:00:00

srun --container-image=nvcr.io#nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi -L
```

Submit it from [Open OnDemand](https://docs.c4e.ai/getting-started/ondemand/)'s Job Composer. The output file should list one GB10 GPU.

The first run takes longer, about 80 seconds, because Slurm imports the container image before it starts. That run also pauses the shared models for everyone for about 3 minutes, as [below](#jobs-pause-the-shared-models) explains. Chat and the API may say the models are paused; they come back on their own.

## Ask for memory

> **Always set --mem**
>
> On the GB10 the CPU and GPU share one pool of memory, and GPU memory counts against your job's limit. Without `--mem`, Slurm gives a job 1 GB per CPU, and GPU work runs out of memory almost at once.

Each Spark has 128 GB in total, and the system keeps some of it. Asking for up to about 100 GB on one node is safe.

## Containers

Jobs run containers through NVIDIA's [Pyxis](https://github.com/NVIDIA/pyxis) and [Enroot](https://github.com/NVIDIA/enroot). Add `--container-image` to `srun`:

- `--container-image=ubuntu:24.04` pulls from Docker Hub.
- `--container-image=nvcr.io#nvidia/pytorch:<tag>` pulls from NVIDIA's NGC catalog. Note the `#` after the registry name.

The Sparks use Arm (aarch64) CPUs, so pick images built for Arm. NVIDIA's NGC images for CUDA and PyTorch include Arm builds.

## Limits

| | |
|---|---|
| Partition | `jobs` (the default) |
| Longest run | 7 days (`--time=7-00:00:00`) |
| GPUs | 1 per node (`--gres=gpu:1`) |
| CPUs | up to 18 or 20 per node |

## Jobs pause the shared models

The models behind [chat](https://docs.c4e.ai/getting-started/chat/) and [the API](https://docs.c4e.ai/getting-started/api/) run as a Slurm job of their own, in an admin-only `serve` partition, on both Sparks. Your jobs come first: when a job in `jobs` needs those nodes, Slurm stops the models and puts them back in the queue. While the cluster has two Sparks, that's almost any job.

> **Your job takes chat and the API offline for everyone**
>
> From the moment your job starts until the models have restarted, chat.c4e.ai and api.c4e.ai can't answer. The models restart by themselves when your job ends, and need a couple of minutes to load. The example job above keeps it offline for about 3 minutes.

While the models are paused, chat answers with a message saying they're paused and come back on their own, and the API returns `503` with `Retry-After: 60`. Nothing is lost: wait a few minutes and try again. The OpenAI SDKs retry the `503` by themselves; in your own code, retry with a delay rather than in a tight loop.

To keep the pause short:

- Ask for the time you need with `--time`, not the 7-day maximum. Slurm ends the job at that limit, so the models come back on time.
- Test with small, short runs before a long one.
- Cancel jobs you no longer want with `scancel <jobid>`.
- Run long jobs when the cluster is quiet. `#SBATCH --begin=22:00` holds a job until 10 PM, and `--begin=now+2hours` works too.

### Check what's running

[Open OnDemand](https://docs.c4e.ai/getting-started/ondemand/)'s **Active Jobs** shows the whole queue. From a shell, run:

```bash
squeue
```

A `tensorfold-serve` job in the `serve` partition marked `R` means the models are running. `PD` means it's waiting for jobs to finish. Your own jobs, and how long they ran:

```bash
squeue --me
sacct -X --format=JobID,JobName,Partition,State,Start,End,Elapsed
```

Once the four new Sparks arrive, the models will move to nodes of their own and jobs will stop pausing them.

Source: https://docs.c4e.ai/getting-started/slurm-jobs/index.mdx
