Two NVIDIA DGX Sparks are online today, linked directly at 200 Gb/s. Four more are coming, and the plan for six puts them on a switched 200 Gb/s fabric with 768 GB of unified memory and about 1.6 TB/s of aggregate memory bandwidth (6 × 273 GB/s). Students reach everything through one sign-in at id.c4e.ai; admins also come in over a private tailnet.
id.c4e.ai: email codes and passkeys. Cloudflare Access checks it for chat, the job portal and Grafana; the API takes keys.chat.c4e.ai: chat, and a personal API key per studentondemand.c4e.ai: files, job composer, active jobs. Agents use mcp.c4e.ai.grafana.c4e.ai with Prometheus and alerts; Tailscale SSHapi.c4e.ai/v1: fixed model names, usage per studentSolid boxes are running today. Dashed boxes are the four Sparks still to arrive and the switches planned for them. Each piece of software has its own page under Stack, starting with Slurm, with a link to its official documentation.
Sign-in
Everyone signs in through Authentik at id.c4e.ai, with a one-time code sent to their email or a passkey. There are no passwords to keep, and nobody installs a VPN. Cloudflare Access checks that sign-in in front of chat.c4e.ai, ondemand.c4e.ai and grafana.c4e.ai. The API at api.c4e.ai skips Access and takes an API key instead, and the agent server at mcp.c4e.ai uses the same Authentik sign-in through OAuth.
Storage and backups
Home folders live on spark-e675 and are shared to every Spark over NFS, so a job sees the same files on any node. Each night the cluster’s own data (accounts, chat history, job records and settings) is backed up to spark-c0a1. Home folders aren’t backed up yet, so keep your own copy of anything you can’t lose.
Monitoring
Admins watch the cluster in Grafana at grafana.c4e.ai, behind the same sign-in. Prometheus collects the Sparks, their GPUs, LiteLLM and the inference engine, and alerts email the admins.
Models
The models run as Slurm jobs on whichever engine is fastest for them. LiteLLM keeps the model names fixed, so an engine swap changes nothing for students or their keys.
Sharing the Sparks
The models run as a Slurm job too, in an admin-only serve partition at the lowest priority. Student and research jobs go to the default jobs partition, which ranks higher. When a job needs the models’ nodes, Slurm preempts them: it stops them and puts them back in the queue (preemption mode REQUEUE). The job starts at once.
While there are only two Sparks, the models span both, so almost any job pauses them. From the moment the job starts until the models have reloaded, chat.c4e.ai and api.c4e.ai can’t answer: chat says the models are paused, and the API returns 503 with Retry-After: 60. When the job ends, the models start again by themselves and need a couple of minutes to load. The example job in the docs keeps it offline for about 3 minutes.
Students can see this with squeue and sacct, and can keep the pause short by asking only for the --time they need or by scheduling long jobs with --begin. Slurm jobs has the details. Once the four new Sparks arrive, the models get nodes of their own and jobs stop pausing them.