previous episode

Reproducible Computational Environments Using Containers: Introduction to Docker and Singularity

next episode

Building Singularity images

Overview

Teaching: 30 min
Exercises: 30 min
Questions

  • How do I create my own Singularity images?

Objectives

  • Understand the different Singularity container file formats.

  • Understand how to build and share your own Singularity containers.

Singularity - Part II

Brief recap

In the first two episodes of this Singularity material we’ve seen how Singularity can be used on a computing platform where you don’t have any administrative privileges. The software was pre-installed and it was possible to work with existing images such as Singularity image files already stored on the platform or images obtained from a remote image repository such as Singularity Hub or Docker Hub.

It is clear that between Singularity Hub and Docker Hub there is a huge array of images available but what if you want to create your own images or customise existing images?

In this first of two episodes in Part II of the Singularity material, we’ll look at building Singularity images.

Preparing to use Singularity for building images

So far you’ve been able to work with Singularity from your own user account as a non-privileged user. This part of the Singularity material requires that you use Singularity in an environment where you have administrative (root) access. While it is possible to build Singularity containers without root access, it is highly recommended that you do this as the root user, as highlighted in this section of the Singularity documentation. Bear in mind that the system that you use to build containers doesn’t have to be the system where you intend to run the containers. If, for example, you are intending to build a container that you can subsequently run on a Linux-based cluster, you could build the container on your own Linux-based desktop or laptop computer. You could then transfer the built image directly to the target platform or upload it to an image repository and pull it onto the target platform from this repository.

There are three different options for accessing a suitable environment to undertake the material in this part of the course:

  1. Run Singularity from within a Docker container - this will enable you to have the required privileges to build images
  2. Install Singularity locally on a system where you have administrative access
  3. Use Singularity on a system where it is already pre-installed and you have administrative (root) access

We’ll focus on the first option in this part of the course. If you would like to install Singularity directly on your system, see the box below for some further pointers. Note that the installation process is an advanced task that is beyond the scope of this course so we won’t be covering this.

Installing Singularity on your local system (optional) [Advanced task]

If you are running Linux and would like to install Singularity locally on your system, Singularity provide the free, open source Singularity Community Edition. You will need to install various dependencies on your system and then build Singularity from source code.

If you are not familiar with building applications from source code, it is strongly recommended that you use the Docker Singularity image, as described below in the “Getting started with the Docker Singularity image” section rather than attempting to build and install Singularity yourself. The installation process is an advanced task that is beyond the scope of this session.

However, if you have Linux systems knowledge and would like to attempt a local install of Singularity, you can find details in the INSTALL.md file within the Singularity repository that explains how to install the prerequisites and build and install the software. Singularity is written in the Go programming language and Go is the main dependency that you’ll need to install on your system. The process of installing Go and any other requirements is detailed in the INSTALL.md file.

Note

If you do not have access to a system with Docker installed, or a Linux system where you can build and install Singularity but you have administrative privileges on another system, you could look at installing a virtualisation tool such as VirtualBox on which you could run a Linux Virtual Machine (VM) image. Within the Linux VM image, you will be able to install Singularity. Again this is beyond the scope of the course.

If you are not able to access/run Singularity yourself on a system where you have administrative privileges, you can still follow through this material as it is being taught (or read through it in your own time if you’re not participating in a taught version of the course) since it will be helpful to have an understanding of how Singularity images can be built.

You could also attempt to follow this section of the lesson without using root and instead using the singularity command’s --fakeroot option. However, you may encounter issues with permissions when trying to build images and run your containers and this is why running the commands as root is strongly recommended and is the approach described in this lesson.

Getting started with the Docker Singularity image

The Singularity Docker image is available from Quay.io.

Familiarise yourself with the Docker Singularity image

  • Using your previously acquired Docker knowledge, get the Singularity image for v3.7.0 and ensure that you can run a Docker container using this image. You might want to use the v3.7.0-slim version of this image since it is significantly smaller than the standard image - the slim version of the image will be used in the examples below.

  • Create a directory (e.g. $HOME/singularity_data) on your host machine that you can use for storage of definition files (we’ll introduce these shortly) and generated image files.

    This directory should be bind mounted into the Docker container at the location /home/singularity every time you run it - this will give you a location in which to store built images so that they are available on the host system once the container exits. (take a look at the -v switch)

Note: To be able to build an image using the Docker Singularity container, you’ll probably need to add the --privileged switch to your docker command line.

Questions:

  • What is happening when you run the container?
  • Can you run an interactive shell in the container?

Running the image

Having a bound directory from the host system accessible within your running Singularity container will give you somewhere to place created images so that they are accessible on the host system after the container exits. Begin by changing into the directory that you created above for storing your definiton files and built images (e.g. $HOME/singularity_data).

You may choose to:

  • open a shell within the Docker image so you can work at a command prompt and run the singularity command directly
  • use the docker run command to run a new container instance every time you want to run the singularity command.

Either option is fine for this section of the material.

Some examples:

To run the singularity command within the docker container directly from the host system’s terminal:

docker run -it --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim cache list

To start a shell within the Singularity Docker container where the singularity command can be run directly:

docker run -it --entrypoint=/bin/sh --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim

To make things easier to read in the remainder of the material, command examples will use the singularity command directly, e.g. singularity cache list. If you’re running a shell in the Docker container, you can enter the commands as they appear. If you’re using the container’s default run behaviour and running a container instance for each run of the command, you’ll need to replace singularity with docker run --privileged -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim or similar.

Building Singularity images

Introduction

As a platform that is widely used in the scientific/research software and HPC communities, Singularity provides great support for reproducibility. If you build a Singularity container for some scientific software, it’s likely that you and/or others will want to be able to reproduce exactly the same environment again. Maybe you want to verify the results of the code or provide a means that others can use to verify the results to support a paper or report. Maybe you’re making a tool available to others and want to ensure that they have exactly the right version/configuration of the code.

Similarly to Docker and many other modern software tools, Singularity follows the “Configuration as code” approach and a container configuration can be stored in a file which can then be committed to your version control system alongside other code. Assuming it is suitably configured, this file can then be used by you or other individuals (or by automated build tools) to reproduce a container with the same configuration at some point in the future.

Different approaches to building images

There are various approaches to building Singularity images. We highlight two different approaches here and focus on one of them:

You can take a look at Singularity’s “Build a Container” documentation for more details on different approaches to building containers.

Why look at Singularity Definition Files?

Why do you think we might be looking at the definition file approach here rather than the sandbox approach?

Discussion

The sandbox approach is great for prototyping and testing out an image configuration but it doesn’t provide the best support for our ultimate goal of reproducibility. If you spend time sitting at your terminal in front of a shell typing different commands to add configuration, maybe you realise you made a mistake so you undo one piece of configuration and change it. This goes on until you have your completed configuration but there’s no explicit record of exactly what you did to create that configuration.

Say your container image file gets deleted by accident, or someone else wants to create an equivalent image to test something. How will they do this and know for sure that they have the same configuration that you had? With a definition file, the configuration steps are explicitly defined and can be easily stored, for example within a version control system, and re-run.

Definition files are small text files while container files may be very large, multi-gigabyte files that are difficult and time consuming to move around. This makes definition files ideal for storing in a version control system along with their revisions.

Creating a Singularity Definition File

A Singularity Definition File is a text file that contains a series of statements that are used to create a container image. In line with the configuration as code approach mentioned above, the definition file can be stored in your code repository alongside your application code and used to create a reproducible image. This means that for a given commit in your repository, the version of the definition file present at that commit can be used to reproduce a container with a known state. It was pointed out earlier in the course, when covering Docker, that this property also applies for Dockerfiles.

We’ll now look at a very simple example of a definition file:

Bootstrap: docker
From: ubuntu:20.04

%post
    apt-get -y update && apt-get install -y python3

%runscript
    python3 -c 'print("Hello World! Hello from our custom Singularity image!")'

A definition file has a number of optional sections, specified using the % prefix, that are used to define or undertake different configuration during different stages of the image build process. You can find full details in Singularity’s Definition Files documentation. In our very simple example here, we only use the %post and %runscript sections.

Let’s step through this definition file and look at the lines in more detail:

Bootstrap: docker
From: ubuntu:20.04

These first two lines define where to bootstrap our image from. Why can’t we just put some application binaries into a blank image? Any applications or tools that we want to run will need to interact with standard system libraries and potentially a wide range of other libraries and tools. These need to be available within the image and we therefore need some sort of operating system as the basis for our image. The most straightforward way to achieve this is to start from an existing base image containing an operating system. In this case, we’re going to start from a minimal Ubuntu 20.04 Linux Docker image. Note that we’re using a Docker image as the basis for creating a Singularity image. This demonstrates the flexibility in being able to start from different types of images when creating a new Singularity image.

The Bootstrap: docker line is similar to prefixing an image path with docker:// when using, for example, the singularity pull command. A range of different bootstrap options are supported. From: ubuntu:20.04 says that we want to use the ubuntu image with the tag 20.04.

Next we have the %post section of the definition file:

%post
    apt-get -y update && apt-get install -y python3

In this section of the file we can do tasks such as package installation, pulling data files from remote locations and undertaking local configuration within the image. The commands that appear in this section are standard shell commands and they are run within the context of our new container image. So, in the case of this example, these commands are being run within the context of a minimal Ubuntu 20.04 image that initially has only a very small set of core packages installed.

Here we use Ubuntu’s package manager to update our package indexes and then install the python3 package along with any required dependencies (in Ubuntu 20.04, the python3 package installs python 3.8.5). The -y switches are used to accept, by default, interactive prompts that might appear asking you to confirm package updates or installation. This is required because our definition file should be able to run in an unattended, non-interactive environment.

Finally we have the %runscript section:

%runscript
    python3 -c 'print("Hello World! Hello from our custom Singularity image!")'

This section is used to define a script that should be run when a container is started based on this image using the singularity run command. In this simple example we use python3 to print out some text to the console.

We can now save the contents of the simple defintion file shown above to a file and build an image based on it. In the case of this example, the definition file has been named my_test_image.def. (Note that the instructions here assume you’ve bound the image output directory you created to the /home/singularity directory in your Docker Singularity container):

$ singularity build /home/singularity/my_test_image.sif /home/singularity/my_test_image.def

Recall from the details at the start of this section that if you are running your command from the host system command line, running an instance of a Docker container for each run of the command, your command will look something like this:

$ docker run -it --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim build /home/singularity/my_test_image.sif /home/singularity/my_test_image.def

The above command requests the building of an image based on the my_test_image.def file with the resulting image saved to the my_test_image.sif file. Note that you will need to prefix the command with sudo if you’re running a locally installed version of Singularity and not running via Docker because it is necessary to have administrative privileges to build the image. You should see output similar to the following:

INFO:    Starting build...
Getting image source signatures
Copying blob da7391352a9b done  
Copying blob 14428a6d4bcd done  
Copying blob 2c2d948710f2 done  
Copying config aa23411143 done  
Writing manifest to image destination
Storing signatures
2020/12/08 09:15:18  info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8
2020/12/08 09:15:19  info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d
2020/12/08 09:15:19  info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601
INFO:    Running post scriptlet
+ apt-get -y update
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
...
  [Package update output truncated]
...
Fetched 16.6 MB in 3s (6050 kB/s)
Reading package lists...
+ apt-get install -y python3
Reading package lists...
...
  [Package install output truncated]
...
Processing triggers for libc-bin (2.31-0ubuntu9.1) ...
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: my_test_image.sif
$

You should now have a my_test_image.sif file in the current directory. Note that in your version of the above output, after it says INFO: Starting build... you may see a series of skipped: already exists messages for the Copying blob lines. This happens when the Docker image slices for the Ubuntu 20.04 image have previously been downloaded and are cached on the system where this example is being run. On your system, if the image is not already cached, you will see the slices being downloaded from Docker Hub when these lines of output appear.

Permissions of the created image file

You may find that the created Singularity image file on your host filesystem is owned by the root user and not your user. In this case, you won’t be able to change the ownership/permissions of the file directly if you don’t have root access.

However, the image file will be readable by you and you should be able to take a copy of the file under a new name which you will then own. You will then be able to modify the permissions of this copy of the image and delete the original root-owned file since the default permissions should allow this.

Testing your Singularity image

In a moment we’ll test the created image on our HPC platform but, first, you should be able to run a shell in an instance of the Docker Singularity container and run your singularity image there.

Run the Singularity image you’ve created

Can you run the Singularity image you’ve just built from a shell within the Docker Singularity container?

Solution

$ docker run -it --entrypoint=/bin/sh --privileged --rm -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.7.0-slim

/ # cd /home/singularity
/home/singularity # singularity run my_test_image.sif

Hello World! Hello from our custom Singularity image!
/home/singularity #

Using singularity run from within the Docker container

It is strongly recommended that you don’t use the Docker container for running Singularity images in any production setting, only for creating them, since the Singularity command runs within the container as the root user.

However, for the purposes of this simple example, the Docker Singularity container provides an ideal environment to test that you have successfully built your container.

Now we’ll test our image on an HPC platform. Move your created .sif image file to a platform with an installation of Singularity. You could, for example, do this using the command line secure copy command scp. For example, the following command would copy my_test_image.sif to the remote server identified by <target hostname> (don’t forget the colon at the end of the hostname!):

$ scp -i <full path to SSH key file> my_test_image.sif <target hostname>:

You could provide a destination path for the file straight after the colon at the end of the above command (without a space), but by default, the file will be uploaded to you home directory.

Try to run the container on the login node of the HPC platform and check that you get the expected output.

More advanced definition files

Here we’ve looked at a very simple example of how to create an image. At this stage, you might want to have a go at creating your own definition file for some code of your own or an application that you work with regularly. There are several definition file sections that were not used in the above example, these are:

The Sections part of the definition file documentation details all the sections and provides an example definition file that makes use of all the sections.

Additional Singularity features

Singularity has a wide range of features. You can find full details in the Singularity User Guide and we highlight a couple of key features here that may be of use/interest:

Remote Builder Capabilities: If you have access to a platform with Singularity installed but you don’t have root access to create containers, you may be able to use the Remote Builder functionality to offload the process of building an image to remote cloud resources. You’ll need to register for a cloud token via the link on the Remote Builder page.

Signing containers: If you do want to share container image (.sif) files directly with colleagues or collaborators, how can the people you send an image to be sure that they have received the file without it being tampered with or suffering from corruption during transfer/storage? And how can you be sure that the same goes for any container image file you receive from others? Singularity supports signing containers. This allows a digital signature to be linked to an image file. This signature can be used to verify that an image file has been signed by the holder of a specific key and that the file is unchanged from when it was signed. You can find full details of how to use this functionality in the Singularity documentation on Signing and Verifying Containers.

Key Points

  • Singularity definition files are used to define the build process and configuration for an image.

  • Singularity’s Docker container provides a way to build images on a platform where Singularity is not installed but Docker is available.

  • Existing images from remote registries such as Docker Hub and Singularity Hub can be used as a base for creating new Singularity images.

previous episode

next episode