Reproducible Computational Environments Using Containers: Introduction to Docker and Singularity (day 1): Creating Containers on the Cloud

Containers can be created on cloud computing platforms

There are lots of ways containers can be created on cloud computing platforms (a.k.a., “in the cloud”). Most commercial cloud providers now offer a container hosting service that will connect to the Docker Hub in order to fetch the container images that you specify, and charge for the time and resources that the containers use. The container hosting will usually be significantly cheaper than full virtualisation services that might be on offer.

Note also that most cloud providers will give you sign-up credit that you can use for free after you first create your account.

For this lesson, though, we instead use an excellent software project repository platform, Bitbucket, that allows users a monthly quota of minutes for which containers of your choice can be run. Bitbucket allows containers to be created in response to the modification of files within your software project.

There are many excellent sites for storing public software repositories

  • Note that Bitbucket, GitHub and GitLab all achieve similar functions.
  • Bitbucket offers container-based features that are easier to get at than the equivalent functions in GitHub, although GitHub will soon catch up when they release their GitHub Actions functionality publicly.

Running a container in the cloud, using your Bitbucket account

Because the ability to use the git version management tool is not a prerequisite of this session, we will use Bitbucket in an atypical manner. However we should still be able to clearly see Bitbucket’s cloud servers running a container of your choice, under your control.

Edit your repository’s bitbucket-pipelines.yml file through the web

You should be looking at a web-based text editor that is headed “bitbucket-pipelines.yml”. The one that I see has 13 lines, all of which are numbered on the left of the text editor.

Bitbucket Pipelines allow you to specify software tools to run, for example, in response to files being changed in your Bitbucket projects. The Bitbucket servers run your software tools within Docker containers, and thus Bitbucket Pipelines can specify Docker images to fetch from the Docker Hub.

Change your repository’s bitbucket-pipelines.yml file to be similar to the following example, but note that you need to replace the Docker Hub user ID (alice in example) with yours. Also, ensure that your indentation steps in line-by-line, the language being used (YAML) gives significance to the indentation of the lines.

image: alice/my-container

pipelines:
  default:
    - step:
        script:
          - /bin/cat /root/my_message

Click the “commit file” button. After you commit your bitbucket-pipelines.yml file, the Bitbucket Pipeline will download the Docker image you specified from the Docker Hub, and display the progress of the computations it runs.

When using the docker run command (as you have done previously), the container takes some default actions after being created, which are specified in your Dockerfile (e.g., the CMD line). Bitbucket Pipelines disable these default actions, instead using the commands listed under the “script:” section in your bitbucket-pipelines.yml. Note that hyphens at the same indentation level are treated as an itemised list. There is only one item in our script: list, namely the command /bin/cat /root/my_message.

If the pipeline runs successfully, a green heading containing a tick icon will be shown near the top of the page. On the right-hand-side of the page, you should see the following headings:

Click on the /bin/cat /root/my_message heading, and you should see that your custom message was shown.

While it is difficult to argue that this container achieves important computational work, you have, nonetheless, demonstrated that Docker images that you create can be run on the cloud. Moreover, many cloud organisations that are willing to create containers from your images will offer generous allowances to you to do so, even if you only have a free account.