Reproducible computational environments using containers: Introduction to Singularity for RSEs

This lesson provides an introduction to using the Singularity container platform. Singularity is particularly suited to running containers on infrastructure where users don’t have administrative privileges, for example shared infrastructure such as High Performance Computing (HPC) clusters.

This lesson will introduce Singularity from scratch showing you how to run a simple container and building up to creating your own containers and running parallel scientific workloads on HPC infrastructure.

Prerequisites

You should have basic familiarity with using a command shell, and the lesson text will at times request that you “open a shell window”, with an assumption that you know what this means. Under Linux or macOS it is assumed that you will access a bash shell (usually the default), using your Terminal application. Under Windows, Powershell or WSL should allow you to use the Unix instructions. The lessons will sometimes request that you use a text editor to create or edit files in particular directories. It is assumed that you either have an editor that you know how to use that runs within the working directory of your shell window (e.g. nano), or that if you use a graphical editor, that you can use it to read and write files into the working directory of your shell.

You should be familiar with logging into remote systems via SSH and copying data to remote systems using “scp” or similar tools.

Software installation

We recommend that you install Singularity/Apptainer locally on your system to participate in this tutorial. While the application is designed to run only on Linux, you install it on macOS via Lima and on Windows via WSL. See the setup page for this lesson for detailed instructions.

Course details

Schedule

Setup Download files required for the lesson
00:00 1. Introducing Containers What are containers, and why might they be useful to me?
00:20 2. Singularity: Getting started What is Singularity and why might I want to use it?
How do I run different commands within a container?
How do I access an interactive shell within a container?
01:10 3. Using Docker images with Singularity How do I use Docker images with Singularity?
01:25 4. Files in Singularity containers How do I make data available in a Singularity container?
What data is made available by default in a Singularity container?
01:45 5. Creating Your Own Container Images How can I make my own Singularity container images?
How do I document the ‘recipe’ for a Singularity container image?
How can I make more complex container images?
03:05 6. Running MPI parallel jobs using Singularity containers How do I set up and run an MPI job from a Singularity container?
04:15 7. Containers in Research Workflows: Reproducibility and Granularity How can I use container images to make my research more reproducible?
How do I incorporate containers into my research workflow?
04:55 8. [Optional] The Singularity cache Why does Singularity use a local cache?
Where does Singularity store images?
05:05 9. [Optional] Using Singularity to run BLAST+ How can I use Singularity to run bioinformatics workflows with BLAST+?
06:05 10. [Optional] Additional topics and next steps How do I understand more on how containers work?
What different container technologies are there and what are differences/implications?
How can I orchestrate different containers?
07:05 Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.