Reproducible computational environments using containers: Introduction to Singularity for RSEs: Glossary

Almost all software depends on other software components to function, but these components have independent evolutionary paths.

Small environments that contain only the software that is needed for a given task are easier to replicate and maintain.

Critical systems that cannot be upgraded, due to cost, difficulty, etc. need to be reproduced on newer systems in a maintainable and self-documented way.

Virtualization allows multiple environments to run on a single computer.

Containerization improves upon the virtualization of whole computers by allowing efficient management of the host computer’s memory and storage resources.

Containers are built from ‘recipes’ that define the required set of software components and the instructions necessary to build/install them within a container image.

Singularity is just one software platform that can create containers and the resources they use.

Singularity is another container platform and it is often used in cluster/HPC/research environments.

Singularity has a different security model to other container platforms, one of the key reasons that it is well suited to HPC and cluster environments.

Singularity has its own container image format (SIF).

The singularity command can be used to pull images from Sylabs Cloud Library and run a container from an image file.

The singularity exec is an alternative to singularity run that allows you to start a container running a specific command.

The singularity shell command can be used to start a container and run an interactive shell within it.

Singularity can start a container from a Docker image which can be pulled directly from Docker Hub.

Your current directory and home directory are usually available by default in a container.

You have the same username and permissions in a container as on the host system.

You can specify additional host system directories to be available in the container.

Singularity recipe files specify what is within Singularity container images.

The singularity build command is used to build a container image from a recipe file.

singularity build requires admin/root privileges so usually needs to be prefixed with sudo.

Singularity allows containers to read and write files from the Singularity host.

You can copy files from your host system into your Singularity container images at creation time by using the %files section in the recipe file.

Singularity images containing MPI applications can be built on one platform and then run on another (e.g. an HPC cluster) if the two platforms have compatible MPI implementations.

When running an MPI application within a Singularity container, use the MPI executable on the host system to launch a Singularity container for each process.

Think about parallel application performance requirements and how where you build/run your image may affect that.

Container images allow us to encapsulate the computation (and data) we have used in our research.

Using online containerimage repositories allows us to easily share computational work we have done.

Using container images along with a DOI service such as Zenodo allows us to capture our work and enables reproducibility.

Singularity caches downloaded images so that an unchanged image isn’t downloaded again when it is requested using the singularity pull command.

You can free up space in the cache by removing all locally cached images or by specifying individual images to remove.

We can use containers to run software without having to install it

The commands we use are very similar to those we would use natively

Singularity handles a lot of complexity around data and internet access for us

TBC