Skip to content

Slurm jobs

Write a Slurm job that asks for a GPU and runs in a container on a DGX Spark.

Every job on the cluster runs through Slurm. Each Spark has one GB10 GPU, so a job asks for at most one GPU per node.

A first GPU job

#!/bin/bash
#SBATCH --job-name=gpu-check
#SBATCH --partition=jobs
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=01:00:00

srun --container-image=nvcr.io#nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi -L

Submit it from Open OnDemand’s Job Composer. The output file should list one GB10 GPU.

Ask for memory

Each Spark has 128 GB in total, and the system keeps some of it. Asking for up to about 100 GB on one node is safe.

Containers

Jobs run containers through NVIDIA’s Pyxis and Enroot. Add --container-image to srun:

  • --container-image=ubuntu:24.04 pulls from Docker Hub.
  • --container-image=nvcr.io#nvidia/pytorch:<tag> pulls from NVIDIA’s NGC catalog. Note the # after the registry name.

The Sparks use Arm (aarch64) CPUs, so pick images built for Arm. NVIDIA’s NGC images for CUDA and PyTorch include Arm builds.

Limits

Partition jobs (the default)
Longest run 7 days (--time=7-00:00:00)
GPUs 1 per node (--gres=gpu:1)
CPUs up to 18 or 20 per node

While the cluster has two Sparks, a running job pauses the shared model that powers chat and the API. Please ask only for the time and resources you need, and cancel jobs you no longer want.

Type to search…

↑↓ navigate↵ selectEsc close