ARCHER2 scheduler: Slurm


Teaching: 30 min
Exercises: 20 min
  • How do I write job submission scripts?

  • How do I control jobs?

  • How do I find out what resources are available?

  • Understand the use of the basic Slurm commands.

  • Know what components make up and ARCHER2 scheduler.

  • Know where to look for further help on the scheduler.

ARCHER2 uses the Slurm job submission system, or scheduler, to manage resources and how they are made available to users. The main commands you will use with Slurm on ARCHER2 are:

Full documentation on Slurm on ARCHER2 can be found in the Running Jobs on ARCHER2 section of the ARCHER2 documentations.

Finding out what resources are available: sinfo

The sinfo command shows the current state of the compute nodes known to the scheduler:

auser@uan01:~> sinfo
standard     up 1-00:00:00    231  alloc nid[001001,001005,001010-001015,001018-001020,001022-001028,001034-001044,001048-001049,001051-001060,001062-001067,001071-001072,001075-001091,001093-001108,001110,001112,001115-001124,001139-001143,001150-001153,001155-001158,001161-001162,001164-001169,001200-001202,001246-001249,001256-001261,001319-001325,001327-001333,001348-001349,001367-001368,001463-001467,001469-001474,001540-001545,001547-001551,001646,001648-001651,001665-001668,001710-001718,001720-001721,001724-001728,001748-001750,001756,001818-001824,001826-001831,001920-001931,001945-001949,001951-001954] 
standard     up 1-00:00:00    669   idle nid[001000,001002-001004,001007-001009,001030-001032,001073,001126-001137,001145-001148,001154,001170,001175-001199,001204-001226,001229-001240,001245,001263-001272,001274-001286,001288-001317,001334-001335,001337-001346,001352-001365,001370-001383,001396-001434,001436-001461,001475-001477,001479-001489,001491-001504,001506-001519,001522-001538,001553-001580,001582-001613,001615-001641,001653-001663,001670-001688,001690-001708,001730-001746,001752-001755,001758-001781,001783-001809,001832,001840-001890,001893-001902,001904-001918,001933-001943,001956-001968,001970-002013,002015-002023] 

There is a row for each node state and partition combination. The default output shows the following columns:

The nodes can be in many different states, the most common you will see are:

If you prefer to see the state of individual nodes, you can use the sinfo -N -l command.

Lots to look at!

Warning! The sinfo -N -l command will produce a lot of output as there are over 5000 individual nodes on ARCHER2!

auser@uan01:~> sinfo -N -l
Thu Oct 29 21:22:08 2020
nid001000      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001001      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001002      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001003      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001004      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001005      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001006      1  standard       down* 256    2:64:2 256000        0      1   (null) Not responding       
nid001007      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001008      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001009      1  standard        idle 256    2:64:2 256000        0      1   (null) none                 
nid001010      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001011      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001012      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001013      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001014      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
nid001015      1  standard   allocated 256    2:64:2 256000        0      1   (null) none                 
...lots of output trimmed...

Explore a compute node

Let’s look at the resources available on the compute nodes where your jobs will actually run. Try running this command to see the name, CPUs and memory available on the worker nodes (the instructors will give you the ID of the compute node to use):

[auser@uan01:~> sinfo -n nid001000 -o "%n %c %m"

This should display the resources available for a standard node. Can you use sinfo to find out the range of node IDs for the high memory nodes?


The high memory nodes have IDs nid000001-nid000004. You can get this by using:

auser@uan01:~> sinfo -N -l -S "-m" | less

The -S "-m" option tells sinfo to print the node list sorted by decreasing memory per node. This output is then piped into less so we can examine the output a page at a time without it scrolling off the screen.

Using batch job submission scripts

Header section: #SBATCH

As for most other scheduler systems, job submission scripts in Slurm consist of a header section with the shell specification and options to the submission command (sbatch in this case) followed by the body of the script that actually runs the commands you want. In the header section, options to sbatch should be prepended with #SBATCH.

Here is a simple example script that runs the xthi program, which shows process and thread placement, across two nodes.

#SBATCH --job-name=my_mpi_job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=128
#SBATCH --cpus-per-task=1
#SBATCH --time=0:10:0
#SBATCH --account=t01
#SBATCH --partition=standard
#SBATCH --qos=standard

# This module needs to be loaded in ALL scripts
module load epcc-job-env

# Now load the "xthi" package
module load xthi


# Load modules, etc.
# srun to launch the executable
srun --cpu-bind=cores xthi

The options shown here are:

We will discuss the srun command further below.

Submitting jobs using sbatch

You use the sbatch command to submit job submission scripts to the scheduler. For example, if the above script was saved in a file called test_job.slurm, you would submit it with:

auser@uan01:~> sbatch test_job.slurm
Submitted batch job 23996

Slurm reports back with the job ID for the job you have submitted

What are the default for sbatch options?

If you do not specify job options, what are the defaults for Slurm on ARCHER2? Submit jobs to find out what the defaults are for:

  1. Budget (or Account) the job is charged to?
  2. Tasks per node?
  3. Number of nodes?
  4. Walltime? (This one is hard!)


(1) Budget: None - fails if submitted without a budget specified

You can get the answers to 2. and 3. this with the following script (once you have realised that you must specify a budget!):

#SBATCH --job-name=my_mpi_job
#SBATCH --account=t01
module load epcc-job-env
echo "Nodes: $SLURM_JOB_NUM_NODES"
echo "Tasks per node: $SLURM_NTASKS_PER_NODE"
module load xthi


srun --cpu-bind=cores xthi

(2) Tasks per node: 256

(3) Number of nodes: 1

Getting the default time limit is more difficult - we need to use sacct to query the time limit set for the job. For example, if the job ID was “12345”, then we could query the time limit with:

auser@uan01:~> sacct -o "TimeLimit" -j 12345

(4) Walltime: Unlimited

Checking progress of your job with squeue

You use the squeue command to show the current state of the queues on ARCHER2. Without any options, it will show all jobs in the queue:

auser@uan01:~> squeue

Cancelling jobs with scancel

You can use the scancel command to cancel jobs that are queued or running. When used on running jobs it stops them immediately.

Getting notified

Slurm on ARCHER2 can also send e-mails to notify you when your job starts, ends, fails, etc. Can you find out how you would setup your job script to send you an e-mail when your job finishes and when it fails? Test your answer, does it work?


The option --mail-type=END,FAIL will send mailings to you when the job ends or fails. You can also use the event TIME_LIMIT to notify you if a job reaches its walltime without finishing and the events TIME_LIMIT_50, TIME_LIMIT_80 and TIME_LIMIT_90 to notify you when your job is 50%, 80% and 90% of the way through the specified walltime.

Running parallel applications using srun

Once past the header section your script consists of standard shell commands required to run your job. These can be simple or complex depending on how you run your jobs but even the simplest job script usually contains commands to:

After this you will usually launch your parallel program using the srun command. At its simplest, srun only needs 1 argument to specify the correct binding of processes to cores (it will use the values supplied to sbatch to work out how many parallel processes to launch). In the example above, our srun command simply looks like:

srun --cpu-bind=cores xthi

Underpopulation of nodes

You may often want to underpopulate nodes on ARCHER2 to access more memory or more memory bandwidth per task. Can you state the sbatch options you would use to run xthi:

  1. On 4 nodes with 64 tasks per node?
  2. On 8 nodes with 2 tasks per node, 1 task per socket?
  3. On 4 nodes with 32 tasks per node, ensuring an even distribution across the 8 NUMA regions on the node?

Once you have your answers run them in job scripts and check that the binding of tasks to nodes and cores output by xthi is what you expect.


  1. --nodes=4 --ntasks-per-node=64
  2. --nodes=8 --ntasks-per-node=2 --ntasks-per-socket=1
  3. --nodes=4 --ntasks-per-node=32 --ntasks-per-socket=16 --cpus-per-task=4

Hybrid MPI and OpenMP jobs

When running hybrid MPI (with the individual tasks also known as ranks or processes) and OpenMP (with multiple threads) jobs you need to leave free cores between the parallel tasks launched using srun for the multiple OpenMP threads that will be associated with each MPI task.

As we saw above, you can use the options to sbatch to control how many parallel tasks are placed on each compute node and can use the --cpus-per-task option to set the stride between parallel tasks to the right value to accommodate the OpenMP threads - the value for --cpus-per-task should usually be the same as that for OMP_NUM_THREADS. To ensure you get the correct thread pinning, you also need to specify an additional OpenMP environment variable and a couple of additional options to srun. Specifically:

As an example, consider the job script below that runs across 2 nodes with 8 MPI tasks per node and 16 OpenMP threads per MPI task (so all 256 cores across both nodes are used).

#SBATCH --job-name=my_hybrid_job
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --time=0:10:0
#SBATCH --account=t01
#SBATCH --partition=standard
#SBATCH --qos=standard

# This module needs to be loaded in ALL scripts
module load epcc-job-env

# Now load the "xthi" package
module load xthi

export OMP_PLACES=cores

# Load modules, etc.
# srun to launch the executable
srun --hint=nomultithread --distribution=block:block xthi

Each ARCHER2 compute node is made up of 8 NUMA (Non Uniform Memory Access) regions (4 per socket) with 16 cores in each region. Programs where the threads of a task span multiple NUMA regions are likely to be less efficient so we recommend using thread counts that fit well into the ARCHER2 compute node layout. Effectively, this means one of the following options for nodes where all cores are used:


STDOUT and STDERR from jobs are, by default, written to a file called slurm-<jobid>.out in the working directory for the job (unless the job script changes this, this will be the directory where you submitted the job). So for a job with ID 12345 STDOUT and STDERR would be in slurm-12345.out.

If you run into issues with your jobs, the Service Desk will often ask you to send your job submission script and the contents of this file to help debug the issue.

If you need to change the location of STDOUT and STDERR you can use the --output=<filename> and the --error=<filename> options to sbatch to split the streams and output to the named locations.

Other useful information

In this section we briefly introduce other scheduler topics that may be useful to users. We provide links to more information on these areas for people who may want to explore these areas more.

Interactive jobs: direct srun

Similar to the batch jobs covered above, users can also run interactive jobs using the srun command directly. srun used in this way takes the same arguments as sbatch but, obviously, these are specified on the command line rather than in a job submission script. As for srun within a batch job, you should also provide the name of the executable you want to run.

For example, to execute xthi across all cores on two nodes (1 MPI task per core and no OpenMP threading) within an interactive job you would issue the following commands:

auser@uan01:~> srun --partition=standard --qos=standard --nodes=2 --ntasks-per-node=128 --cpus-per-task=1 --time=0:10:0 --account=t01 xthi
Node    0, hostname nid001030
Node    1, hostname nid001031
Node    0, rank    0, thread   0, (affinity = 0,128)
Node    0, rank    1, thread   0, (affinity = 16,144)
Node    0, rank    2, thread   0, (affinity = 32,160)
Node    0, rank    3, thread   0, (affinity = 48,176)
Node    0, rank    4, thread   0, (affinity = 64,192)
Node    0, rank    5, thread   0, (affinity = 80,208)
Node    0, rank    6, thread   0, (affinity = 96,224)
Node    0, rank    7, thread   0, (affinity = 112,240)
Node    0, rank    8, thread   0, (affinity = 1,129)
Node    0, rank    9, thread   0, (affinity = 17,145)
Node    0, rank   10, thread   0, (affinity = 33,161)
Node    0, rank   11, thread   0, (affinity = 49,177)
Node    0, rank   12, thread   0, (affinity = 65,193)
Node    0, rank   13, thread   0, (affinity = 81,209)
Node    0, rank   14, thread   0, (affinity = 97,225)
...long output trimmed...

Key Points

  • ARCHER2 uses the Slurm scheduler.

  • srun is used to launch parallel executables in batch job submission scripts.

  • There are a number of different partitions (queues) available.