Skip to content

Architecture

How the C4E cluster fits together, from sign-in to the DGX Sparks and the links between them.

Two NVIDIA DGX Sparks are online today, linked directly at 200 Gb/s. Four more are coming, and the plan for six puts them on a switched 200 Gb/s fabric with 768 GB of unified memory and about 1.6 TB/s of aggregate memory bandwidth (6 × 273 GB/s). Students reach everything through one sign-in at id.c4e.ai; admins also come in over a private tailnet.

OnlineComing or planned

Solid boxes are running today. Dashed boxes are the four Sparks still to arrive and the switches planned for them. Each piece of software has its own page under Stack, starting with Slurm, with a link to its official documentation.

Sign-in

Everyone signs in through Authentik at id.c4e.ai, with a one-time code sent to their email or a passkey. There are no passwords to keep, and nobody installs a VPN. Cloudflare Access checks that sign-in in front of chat.c4e.ai, ondemand.c4e.ai and grafana.c4e.ai. The API at api.c4e.ai skips Access and takes an API key instead, and the agent server at mcp.c4e.ai uses the same Authentik sign-in through OAuth.

Storage and backups

Home folders live on spark-e675 and are shared to every Spark over NFS, so a job sees the same files on any node. Each night the cluster’s own data (accounts, chat history, job records and settings) is backed up to spark-c0a1. Home folders aren’t backed up yet, so keep your own copy of anything you can’t lose.

Monitoring

Admins watch the cluster in Grafana at grafana.c4e.ai, behind the same sign-in. Prometheus collects the Sparks, their GPUs, LiteLLM and the inference engine, and alerts email the admins.

Models

The models run as Slurm jobs on whichever engine is fastest for them. LiteLLM keeps the model names fixed, so an engine swap changes nothing for students or their keys.

Sharing the Sparks

The models run as a Slurm job too, in an admin-only serve partition at the lowest priority. Student and research jobs go to the default jobs partition, which ranks higher. When a job needs the models’ nodes, Slurm preempts them: it stops them and puts them back in the queue (preemption mode REQUEUE). The job starts at once.

While there are only two Sparks, the models span both, so almost any job pauses them. From the moment the job starts until the models have reloaded, chat.c4e.ai and api.c4e.ai can’t answer: chat says the models are paused, and the API returns 503 with Retry-After: 60. When the job ends, the models start again by themselves and need a couple of minutes to load. The example job in the docs keeps it offline for about 3 minutes.

Students can see this with squeue and sacct, and can keep the pause short by asking only for the --time they need or by scheduling long jobs with --begin. Slurm jobs has the details. Once the four new Sparks arrive, the models get nodes of their own and jobs stop pausing them.

Type to search…

↑↓ navigate↵ selectEsc close