Singularity: Getting started
OverviewTeaching: 30 min
Exercises: 20 minQuestions
What is Singularity and why might I want to use it?
How do I run different commands within a container?
How do I access an interactive shell within a container?Objectives
Understand what Singularity is and when you might want to use it.
Undertake your first run of a simple Singularity container.
Learn how to run different commands when starting a container.
Learn how to open an interactive shell within a container environment.
The episodes in this lesson will introduce you to the Singularity container platform and demonstrate how to set up and use Singularity.
Work in progress…
This lesson is new material that is under ongoing development. We will introduce Singularity and demonstrate how to work with it. As the tools and best practices continue to develop, elements of this material are likely to evolve. We welcome any comments or suggestions on how the material can be improved or extended.
What is Singularity?
You may be familiar with Docker, another container platform that is now used widely. If you are, you will see that in some ways, Singularity is similar to Docker. However, in other ways, particularly in terms of the system’s architecture, it is fundamentally different. These differences mean that Singularity is particularly well-suited to running on shared platforms such as distributed, High Performance Computing (HPC) infrastructure, as well as on a Linux laptop or desktop.
Singularity runs containers from container images which, as we discussed, are essentially a virtual computer disk that contains all of the necessary software, libraries and configuration to run one or more applications or undertake a particular task, e.g. to support a specific research project. This saves you the time and effort of installing and configuring software on your own system or setting up a new computer from scratch, as you can simply run a Singularity container from an image and have a virtual environment that is equivalent to the one used by the person who created the image. Singularity/Apptainer is increasingly widely used in the research community for supporting research projects due to its support for shared computing platforms.
System administrators will not, generally, install Docker on shared computing platforms such as lab desktops, research clusters or HPC platforms because the design of Docker presents potential security issues for shared platforms with multiple users. Singularity/Apptainer, on the other hand, can be run by end-users entirely within “user space”, that is, no special administrative privileges need to be assigned to a user in order for them to run and interact with containers on a platform where Singularity has been installed.
A little history…
Singularity is open source software and was initially developed within the research community. A couple of years ago, the project was “forked” something that is not uncommon within the open source software community, with the software effectively splitting into two projects going in different directions. The fork is being developed by a commercial entity, Sylabs.io who provide both the free, open source SingularityCE (Community Edition) and Pro/Enterprise editions of the software. The original open source Singularity project has recently been renamed to Apptainer and has moved into the Linux Foundation. The initial release of Apptainer was made about a year ago, at the time of writing. While earlier versions of this course focused on versions of Singularity released before the project fork, we now base the course material on recent Apptainer releases. Despite this, the basic features of Apptainer/Singularity remain the same and so this material is equally applicable whether you’re working with a recent Apptainer release or a slightly older Singularity version. Nonetheless, it is useful to be aware of this history and that you may see both Singularity and Apptainer being used within the research community over the coming months and years.
Another point to note is that some systems that have a recent Apptainer release
installed may also provide a
singularity command that is simply a link to the
apptainer executable on the system. This helps to ensure that existing
scripts being used on the system that were developed before the migration to
Apptainer will still function correctly.
For now, the remainder of this material refers to Singularity but where you
have a release of Apptainer installed on your local system, you can simply
replace references to
apptainer, if you wish.
Open a terminal on the system that you are using for the course and check that
singularity command is available in your terminal:
$ singularity --version
singularity-ce version 3.11.0
Depending on the version of Singularity installed on your system, you may see a different version.
Loading a module
HPC systems often use modules to provide access to software on the system so you may need to use the command:
$ module load singularity
before you can use the
singularitycommand on remote systems. However, this depends on how the system is configured. If in doubt, consult the documentation for the system you are using or contact the support team.
Images and containers: reminder
A quick reminder on terminology: we refer to both container images and containers. What is the difference between these two terms?
Container images (sometimes just images) are bundles of files including an operating system, software and potentially data and other application-related files. They may sometimes be referred to as a disk image or image and they may be stored in different ways, perhaps as a single file, or as a group of files. Either way, we refer to this file, or collection of files, as an image.
A container is a virtual environment that is based on a container image. That is, the files, applications, tools, etc that are available within a running container are determined by the image that the container is started from. It may be possible to start multiple container instances from an image. You could, perhaps, consider an image to be a form of template from which running container instances can be started.
Getting a container image and running a Singularity container
Singularity uses the Singularity Image Format (SIF) and container images are provided as single
SIF files (usually with a
.img filename extension). Singularity container images can be pulled from the Sylabs Cloud Library, a registry for Singularity container images. Singularity is also capable of running containers based on container images pulled from Docker Hub and other Docker image repositories (e.g. Quay.io). We will look at accessing container images from Docker Hub later in the course.
Sylabs Remote Builder
Note that in addition to providing a repository that you can pull container images from, Sylabs Cloud Library can also build Singularity images for you from a recipe - a configuration file defining the steps to build an image. We will look at recipes and building images later in the workshop.
Pulling a container image from a remote library
Let’s begin by creating a
test directory, changing into it and pulling an existing container image from a remote library:
$ mkdir test
$ cd test
$ singularity pull hello-world.sif docker://hello-world
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 70f5ac315c5a done
Copying config 286cc18214 done
Writing manifest to image destination
2023/09/06 14:59:34 info unpack layer: sha256:70f5ac315c5af948332962ff4678084ebcc215809506e4b8cd9e509660417205
INFO: Creating SIF file...
What just happened? We pulled a container image from a remote repository (in this case, stored on Docker Hub) using the
singularity pull command and directed it to store the container image in a file using the name
hello-world.sif in the current directory. If you run the
ls command, you should see that the
hello-world.sif file is now present in the current directory.
Why is the protocol
The OCI format has become the defacto standard for distributing containers - this has evolved from the format Docker originally developed. We need to tell Apptainer when we want to convert remote containers that are saved in OCI format into its native format - we do that using the
$ ls -lh
-rwxr-xr-x. 1 auser group 45056 Sep 6 15:00 hello-world.sif
Running a Singularity container
We can now run a container based on the
hello-world.sif container image:
$ singularity run hello-world.sif
WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
For more examples and ideas, visit:
The above command ran a lolcow container based on the container image we downloaded from the online repository and the resulting output was shown.
You may see a warning such as:
INFO: underlay of /etc/localtime required more than 50 (77) bind mounts
We will explain
mountsand what this info statement means later in the workshop
What just happened? When we use the
singularity run command, Singularity does three things:
|1. Starts a Running Container
|2. Performs Default Action
|3. Shuts Down the Container
|Starts a running container, based on the container image. Think of this as the “alive” or “inflated” version of the container – it’s actually doing something.
|If the container has a default action set, it will perform that default action. This could be as simple as printing a message (as above) or running a whole analysis pipeline!
|Once the default action is complete, the container stops running (or exits).
How did the container determine what to do when we ran it? What did running the container actually do to result in the displayed output?
When you run a container from a Singularity container image using the
singularity run command, the container runs the default run script that is embedded within the container image. This is a shell script that can be used to run commands, tools or applications stored within the container image on container startup. We can inspect the container image’s run script using the
singularity inspect command:
$ singularity inspect -r hello-world.sif
This shows us the script within the
hello-world.sif image configured to run by default when we use the
singularity run command.
Running specific commands within a container
We now know that we can use the
singularity inspect command to see the run script that a container is configured to run by default. What if we want to run a different command within a container?
If we know the path of an executable that we want to run within a container, we can use the
singularity exec command. For example, using a container based on the lightweight Alpine Linux distribution which we can pull from Docker Hub we can run the following to first create a Singularity container image file from the image on Docker Hub and then print a message from a running container:
singularity pull alpine.sif docker://alpine
singularity exec alpine.sif echo "Hello, world"
Here we see that a container has been started from the
alpine.sif image and the
echo command has been run within the container, passing the input
Hello, world. The command has echoed the provided input to the console and the container has terminated.
Note that the use of
singularity exec has overriden any run script set within the image metadata and the command that we specified as an argument to
singularity exec has been run instead.
Basic exercise: Running a different command within the “hello-world” container
Can you run a container based on the
alpine.sifimage that prints the current date and time?
singularity exec alpine.sif /bin/date
Fri Jun 26 15:17:44 BST 2020
singularity run and
Above, we used the
singularity exec command. In earlier episodes of this
course we used
singularity run. To clarify, the difference between these
two commands is:
singularity run: This will run the default command set for containers based on the specified image. This default command is set within the image metadata when the image is built (we’ll see more about this in later episodes). You do not specify a command to run when using
singularity run, you simply specify the image file name. As we saw earlier, you can use the
singularity inspectcommand to see what command is run by default when starting a new container based on an image.
singularity exec: This will start a container based on the specified image and run the command provided on the command line following
singularity exec <image file name>. This will override any default command specified within the image metadata that would otherwise be run if you used
Opening an interactive shell within a container
If you want to open an interactive shell within a container, Singularity provides the
singularity shell command. Again, using the
alpine.sif image, and within our
test directory, we can run a shell within a container from the hello-world image:
$ singularity shell alpine.sif
As shown above, we have opened a shell in a new container started from the
alpine.sif image. Note that the shell prompt has changed to show we are now within the Singularity container.
exit command to exit from the container shell.
Singularity is another container platform and it is often used in cluster/HPC/research environments.
Singularity has a different security model to other container platforms, one of the key reasons that it is well suited to HPC and cluster environments.
Singularity has its own container image format (SIF).
singularitycommand can be used to pull images from Sylabs Cloud Library and run a container from an image file.
singularity execis an alternative to
singularity runthat allows you to start a container running a specific command.
singularity shellcommand can be used to start a container and run an interactive shell within it.