previous episode

Introduction to High-Performance Computing

next episode

Using resources effectively

Overview

Teaching: 10 min
Exercises: 30 min
Questions

  • How do we monitor our jobs?

  • How can I get my jobs scheduled more easily?

Objectives

  • Understand how to look up job statistics and profile code.

  • Understand job size implications.

We’ve touched on all the skills you need to interact with an HPC cluster: logging in over SSH, loading software modules, submitting parallel jobs, and finding the output. Let’s learn about estimating resource usage and why it might matter. To do this we need to understand the basics of benchmarking. Benchmarking is essentially performing simple experiments to help understand how the performance of our work varies as we change the properties of the jobs on the cluster - including input parameters, job options and resources used.

Our example

In the rest of this episode, we will use an example parallel application that sharpens an input image. Although this is a toy problem, it exhibits all the properties of a full parallel application that we are interested in for this course.

The main resource we will consider here is the use of compute core time as this is the resource you are usually charged for on HPC resources. However, other resources - such as memory use - may also have a bearing on how you choose resources and constrain your choice.

For those that have come across HPC benchmarking before, you may be aware that people often make a distinction between strong scaling and weak scaling:

  • Strong scaling is where the problem size (i.e. the application) stays the same size and we try to use more cores to solve the problem faster.
  • Weak scaling is where the problem size increases at the same rate as we increase the core count so we are using more cores to solve a larger problem.

Both of these approaches are equally valid uses of HPC. This example looks at strong scaling.

Before we work on benchmarking, it is useful to define some terms for the example we will be using

Accessing the software and input

The sharpen program has been precompiled on ARCHER2, you can add it to your PATH variable with the commands:

export PATH=/work/ta076/ta076/shared/bin:$PATH
export FUZZY_INPUT=/work/ta076/ta076/shared/fuzzy.pgm

Once you have set the required environment variables, you can access the program as sharpen-mpi. You will also need to get a copy of the input file for this application. To do this, copy it from the central install location to your directory with (note you must have loaded the sharpen module as described above for this to work):

 cp $FUZZY_INPUT .

Baseline: running in serial

Before starting to benchmark an application to understand what resources are best to use, you need a baseline performance result. In more formal benchmarking, your baseline is usually the minimum number of cores or nodes you can run on. However, for understanding how best to use resources, as we are doing here, your baseline could be the performance on any number of cores or nodes that you can measure the change in performance from.

Our sharpen application is small enough that we can run a serial (i.e. using a single core) job for our baseline performance so that is where we will start

Run a single core job

Write a job submission script that runs the sharpen application on a single core. You will need to take an initial guess as to the walltime to request to give the job time to complete. Submit the job and check the contents of the STDOUT file to see if the application worked or not.

Solution

Creating a file called submit_sharpen.slurm:

#!/bin/bash

#SBATCH --partition=standard
#SBATCH --qos=short
#SBATCH --reservation=shortqos

#SBATCH --job-name=sharpen
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --time=00:15:00

export PATH=/work/ta076/ta076/shared/bin:$PATH
export FUZZY_INPUT=/work/ta076/ta076/shared/fuzzy.pgm

cp $FUZZY_INPUT .

srun --hint=nomultithread --distribution=block:block sharpen-mpi

Run application using a single process (i.e. in serial) with a blocking srun command:

srun --hint=nomultithread --distribution=block:block sharpen-mpi

Submit with to the batch queue with:

userid@uan01:/work/ta076/ta076/userid> sbatch  submit_sharpen.slurm

Output in the job log should look something like:

Input file is: fuzzy.pgm
Image size is 564 x 770
Using a filter of size 17 x 17
Reading image file: fuzzy.pgm
... done
Starting calculation ...
On core 0
... finished
Writing output file: sharpened.pgm
... done
Calculation time was 5.400482 seconds
Overall run time was 5.496556 seconds

Once your job has run, you should look in the output to identify the performance. Most HPC programs should print out timing or performance information (usually somewhere near the bottom of the summary output) and sharpen is no exception. You should see two lines in the output that look something like:

Calculation time was 5.579000 seconds
Overall run time was 5.671895 seconds

You can also get an estimate of the overall run time from the final job statistics. If we look at how long the finished job ran for, this will provide a quick way to see roughly what the runtime was. This can be useful if you want to know quickly if a job was faster or not than a previous job (as you do not have to find the output file to look up the performance) but the number is not as accurate as the performance recorded by the application itself and also includes static overheads from running the job (such as loading modules and startup time) that can skew the timings. To do this on use sacct -l -j with the job ID, e.g.:

userid@uan01:/work/ta076/ta076/userid> sacct -l -j 12345

JOBID USER         ACCOUNT     NAME           ST REASON START_TIME         T...
36856 yourUsername yourAccount example-job.sh R  None   2017-07-01T16:47:02 ...

Viewing the sharpened output image

To see the effect of the sharpening algorithm, you can view the images using the display program from the ImageMagick suite.

display sharpened.pgm

Type q in the image window to close the program. To view the image you will need an X window client installed and you will have to have logged into with the ssh -Y option to export the display back to your local system. If you are using Windows, the MobaXterm program provides a login shell with X capability. If you are using macOS, then you will need to install XQuartz. If you are using Linux then X should just work!

Running in parallel and benchmarking performance

We have now managed to run the sharpen application using a single core and have a baseline performance we can use to judge how well we are using resources on the system.

Note that we also now have a good estimate of how long the application takes to run so we can provide a better setting for the walltime for future jobs we submit. Lets now look at how the runtime varies with core count.

Benchmarking the parallel performance

Modify your job script to run on multiple cores and evaluate the performance of sharpen on a variety of different core counts and use multiple runs to complete a table like the one below.

If you examine the log file you will see that it contains two timings: the total time taken by the entire program (including IO) and the time taken solely by the calculation. The image input and output is not parallelised so this is a serial overhead, performed by a single processor. The calculation part is, in theory, perfectly parallel (each processor operates on different parts of the image) so this should get faster on more cores. The IO time in the table below is the difference between the calculation time and the overall run time; the Calculation core seconds is the calculation time multiplied by the number of cores.

Cores Overall run time (s) Calculation time (s) IO time (s) Calculation core seconds
1 (serial)        
2        
4        
8        
16        
32        
64        
128        
256        

Look at your results – do they make sense? Given the structure of the code, you would expect the IO time to be roughly constant, and the performance of the calculation to increase linearly with the number of cores: this would give a roughly constant figure for the Calculation core seconds. Is this what you observe?

Solution

The table below shows example timings for runs on

Cores Overall run time (s) Calculation time (s) IO time (s) Calculation core seconds
1 3.188 2.890 0.298 2.890
2 1.697 1.432 0.265 2.864
4 1.068 0.716 0.352 2.864
8 0.717 0.372 0.345 2.976
16 0.516 0.181 0.335 2.896
32 0.393 0.093 0.300 2.976
64 0.412 0.053 0.359 3.392
128 0.317 0.030 0.287 3.840
256 0.312 0.018 0.294 4.608

Understanding the performance

Now we have some data showing the performance of our application we need to try and draw some useful conclusions as to what the most efficient set of resources are to use for our jobs. To do this we introduce two metrics:

We will now use our performance results to compute these two metrics for the sharpen application and use the metrics to evaluate the performance and make some decisions about the most effective use of resources.

Computing the speedup and parallel efficiency

Use your Overall run times from above to fill in a table like the one below.

Cores Overall run time (s) Ideal speedup Actual speedup Parallel efficiency
1 (serial)        
2        
4        
8        
16        
32        
64        
128        
256        

Given your results, try to answer the following questions:

  1. What is the core count where you get the most efficient use of resources, irrespective of run time?
  2. What is the core count where you get the fastest solution, irrespective of efficiency?
  3. What do you think a good core count choice would be for this application that balances time to solution and efficiency? Why did you choose this option?

Solution

The table below gives example results for based on the example runtimes given in the solution above.

Cores Overall run time (s) Ideal speedup Actual speedup Parallel efficiency
1 3.188 1.000 1.000 1.000
2 1.697 2.000 1.879 0.939
4 1.068 4.000 2.985 0.746
8 0.717 8.000 4.446 0.556
16 0.516 16.000 6.178 0.386
32 0.393 32.000 8.112 0.253
64 0.412 64.000 7.738 0.121
128 0.317 128.000 10.057 0.079
256 0.312 256.000 10.218 0.040

What is the core count where you get the most efficient use of resources?

Just using a single core is the cheapest (and always will be unless your speedup is better than perfect – “super-linear” speedup). However, it may not be possible to run on small numbers of cores depending on how much memory you need or other technical constraints.

Note: on most high-end systems, nodes are not shared between users. This means you are charged for all the CPU-cores on a node regardless of whether you actually use them. Typically we would be running on many hundreds of CPU-cores not a few tens, so the real question in practice is: what is the optimal number of nodes to use?

What is the core count where you get the fastest solution, irrespective of efficiency?

256 cores gives the fastest time to solution.

The fastest time to solution does not often make the most efficient use of resources so to use this option, you may end up wasting your resources. Sometimes, when there is time pressure to run the calculations, this may be a valid approach to running applications.

What do you think a good core count choice would be for this application to use?

4 cores is probably a good number of cores to use with a parallel efficiency of 75%.

Usually, the best choice is one that delivers good parallel efficiency with an acceptable time to solution. Note that acceptable time to solution differs depending on circumstances so this is something that the individual researcher will have to assess. Good parallel efficiency is often considered to be 70% or greater though many researchers will be happy to run in a regime with parallel efficiency greater than 60%. As noted above, running with worse parallel efficiency may also be useful if the time to solution is an overriding factor.

Tips

Here are a few tips to help you use resources effectively and efficiently on HPC systems:

Key Points

  • The smaller your job, the faster it will schedule.

previous episode

next episode