Singularity

This page details two methods for building a Singularity container with Firedrake. Both methods rely on converting a Docker container, either the one generated by Firedrake's own CI, or one built from scratch. Using the CI container is easier and simpler, and it is recommended to try this first. However, the CI container will not always be sufficient/compatible, e.g. if you need a specific MPI version or a particular PETSc package not included in the CI container. If this is the case, then you can build a Docker container from scratch and convert that to singularity.

For actually running a Singularity container on a particular HPC system, please see the sidebar or the system's own documentation. If the sidebar doesn't have a page for running Singularity on the system you are using, please consider creating a page to record your workflow for others.

Convert the CI Docker container

A two step procedure is required for performing the image conversion. This example uses the firedrake-vanilla container, but firedrake, firedrake-notebooks and firedrake-complex should also work correctly.

1. Build the image in a sandbox from the Dockerhub singularity build --sandbox ./firedrake-vanilla docker://firedrakeproject/firedrake-vanilla

2. Convert the sandboxed image to a Singularity container singularity build firedrake-vanilla.sif ./firedrake-vanilla

Create a Docker container from scratch.

This is a four step procedure. First, build a Docker container with PETSc, the secondly build a Docker container with Firedrake from the PETSc container. These steps closely follow the Dockerfiles used by the CI. The third and fourth steps are just the two steps listed above for the CI container, except converting your own container.

1. Create a Docker container with a minimal PETSc installation using the following Dockerfile (arguments explained below). This Dockerfile is based heavily off the Dockerfile.env used by the CI, except it only builds a real, no-debug PETSc. Modifying it to build complex, debug, SLEPc as well or instead should be straightforward.

# DockerFile for an environment into which firedrake can be installed.                                                                                                                        

# setup the base OS
FROM ubuntu:22.04
USER root
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

# What is this container?
LABEL name="petsc-environment", \
      description="A PETSc build with a minimal set of packages for Firedrake"

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# User inputs: build variables

# compiler flags
ARG BUILD_ARCH="native"
ARG BUILD_COPTFLAGS='-O3 -march=$BUILD_ARCH -mtune=$BUILD_ARCH'
ARG BUILD_CXXOPTFLAGS='-O3 -march=$BUILD_ARCH -mtune=$BUILD_ARCH'
ARG BUILD_FOPTFLAGS='-O3 -march=$BUILD_ARCH -mtune=$BUILD_ARCH'

# how many cores can make use
ARG BUILD_MAKE_NP=8

# debug build?
ARG BUILD_DEBUG=0

# what mpi to use
ARG MPICH_DOWNLOAD_VERSION="4.1.2"
ARG MPICH_DOWNLOAD_DEVICE="ch3:sock"

# extra packages for petsc to install e.g. netcdf, suitesparse, ...
ARG EXTRA_PACKAGES=""

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# essential packages installed on every build
ENV CORE_PACKAGES=" \
   hdf5 \
   hwloc \
   mumps \
   ptscotch \
   scalapack"

LABEL build.debug=${BUILD_DEBUG} \
      build.mpi.version=${MPICH_DOWNLOAD_VERSION} \
      build.mpi.device=${MPICH_DOWNLOAD_DEVICE} \
      build.extra-packages=${EXTRA_PACKAGES}

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Update and install required packages for Firedrake
RUN apt-get update \
    && apt-get -y dist-upgrade \
    && DEBIAN_FRONTEND=noninteractive apt-get -y install tzdata \
    && apt-get -y install curl vim docker.io \
                 openssh-client build-essential autoconf automake \
                 cmake gfortran git libopenblas-serial-dev \
                 libtool python3-dev python3-pip python3-tk python3-venv \
                 python3-requests zlib1g-dev libboost-dev sudo gmsh \
                 bison flex \
                 liboce-ocaf-dev \
                 swig graphviz \
                 libcurl4-openssl-dev libxml2-dev \
    && rm -rf /var/lib/apt/lists/*

# Use a more sane locale
ENV LC_ALL C.UTF-8

# Set up user so that we do not run as root
RUN useradd -m -s /bin/bash -G sudo firedrake && \
    echo "firedrake:docker" | chpasswd && \
    echo "firedrake ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers && \
    ldconfig

USER firedrake
WORKDIR /home/firedrake

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Fetch PETSc
RUN git clone https://github.com/firedrakeproject/petsc.git

# Build MPICH manually because we don't want PETSc to build it twice
RUN bash -c 'cd petsc; \
    export MPICH_DOWNLOAD_URL=https://github.com/pmodels/mpich/releases/download/v${MPICH_DOWNLOAD_VERSION}/mpich-${MPICH_DOWNLOAD_VERSION}.tar.gz; \
    DOWNLOAD_PACKAGES=" "; \
    for pkg in $CORE_PACKAGES $EXTRA_PACKAGES; \
        do DOWNLOAD_PACKAGES+="--download-$pkg "; done; \
    ./configure \
        --COPTFLAGS=$BUILD_COPTFLAGS \
        --CXXOPTFLAGS=$BUILD_CXXOPTFLAGS \
        --FOPTFLAGS=$BUILD_FOPTFLAGS \
        --with-c2html=0 \
        --with-debugging=$BUILD_DEBUG \
        --with-fortran-bindings=0 \
        --with-make-np=$BUILD_MAKE_NP \
        --with-shared-libraries=1 \
        --with-zlib \
        --download-mpich=$MPICH_DOWNLOAD_URL \
        --download-mpich-device=$MPICH_DOWNLOAD_DEVICE \
        $DOWNLOAD_PACKAGES \
        PETSC_ARCH=packages; \                                                                                                                                                                
        mv packages/include/petscconf.h packages/include/old_petscconf.nope;'
# Don't run make here, we only want MPICH and HWLOC
# It is also necessary to move `petscconf.h` so packages isn't treated like a working PETSc

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Build default Firedrake PETSc
RUN bash -c 'cd petsc; \
    export PACKAGES=/home/firedrake/petsc/packages; \
    DOWNLOAD_PACKAGES_DIR=" "; \
    for pkg in $CORE_PACKAGES $EXTRA_PACKAGES; \
        do DOWNLOAD_PACKAGES_DIR+="--with-$pkg-dir=$PACKAGES "; done; \
    ./configure \
        --COPTFLAGS=$BUILD_COPTFLAGS \
        --CXXOPTFLAGS=$BUILD_CXXOPTFLAGS \
        --FOPTFLAGS=$BUILD_FOPTFLAGS \
        --with-c2html=0 \
        --with-debugging=$BUILD_DEBUG \
        --with-fortran-bindings=0 \
        --with-make-np=$BUILD_MAKE_NP \
        --with-shared-libraries=1 \
        --with-bison \
        --with-flex \
        --with-zlib \
        --with-mpi-dir=$PACKAGES \
        $DOWNLOAD_PACKAGES_DIR \
        PETSC_ARCH=default; \
    make PETSC_DIR=/home/firedrake/petsc PETSC_ARCH=default all;'

RUN bash -c 'cd petsc; \
   make PETSC_DIR=/home/firedrake/petsc PETSC_ARCH=default check;'

# Clean up unnecessary files
RUN rm -rf /home/firedrake/petsc/**/externalpackages \
    && rm -rf /home/firedrake/petsc/src/docs \
    && rm -f /home/firedrake/petsc/src/**/tutorials/output/* \
    && rm -f /home/firedrake/petsc/src/**/tests/output/*

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Set some useful environment variables
ENV PETSC_ARCH=default \
    PETSC_DIR=/home/firedrake/petsc \
    MPICH_DIR=/home/firedrake/petsc/packages/bin \
    HDF5_DIR=/home/firedrake/petsc/packages \
    HDF5_MPI=ON \
    OMP_NUM_THREADS=1 \
    OPENBLAS_NUM_THREADS=1

This Dockerfile has various command line arguments, all of which are defaulted. These defaults may or may not be sensible depending on the system you are running on. You should check them all yourself to make sure. The arguments are:

• BUILD_ARCH: Argument for the compiler flags -march and -mtune.
• BUILD_{C,CXX,F}OPTFLAGS: Compiler optimisation flags.
• BUILD_MAKE_NP: Number of processes used by make.
• BUILD_DEBUG: Whether to build in debug (1) or production (0) mode.
• MPICH_DOWNLOAD_VERSION: The MPICH version to download, e.g. "4.1.2".
• MPICH_DOWNLOAD_DEVICE: The MPICH device to use e.g. "ch3:sock". See the MPICH documentation for more details.
• EXTRA_PACKAGES: Extra PETSc packages to download in addition to the minimal set.

Arguments to pay particular attention to include BUILD_ARCH (the default native is not appropriate if you are not building on the target system), MPICH_DOWNLOAD_VERSION (make sure the target system supports the MPI version used by MPICH), and MPICH_DOWNLOAD_DEVICE (ch3:sock should work almost anywhere, but is not very optimised).

The Dockerfile is invoked like so, with an example of how to provide arguments:

docker build \
   --build-arg BUILD_ARCH="x86-64" \
   --build-arg MPICH_DOWNLOAD_DEVICE="ch3:nemesis" \
   --build-arg EXTRA_PACKAGES="netcdf suitesparse" \
   --tag=petsc-env \
   --file=dockerfile.petsc-env .

This creates a Docker image called petsc-env in your local registry.

2. Create a Docker container with a Firedrake installation using the following Dockerfile. This Dockerfile is again based heavily off the Dockerfile.vanilla used in the CI.

# DockerFile for a plain Firedrake suitable for testing Firedrake components and applications

FROM petsc-env

# What is this container?
LABEL name="firedrake", \
      description="A minimal Firedrake installation"

# Set this to 'master' to use current firedrake.
# Set this to a different branch to build from
# there, which can include modifications to the
# install script.
ARG FIREDRAKE_BRANCH=master

LABEL firedrake.branch=${FIREDRAKE_BRANCH}

USER firedrake
WORKDIR /home/firedrake

# Copy the firedrake-install from the correct branch
RUN curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/${FIREDRAKE_BRANCH}/scripts/firedrake-install                                                                     
RUN bash -c 'python3 firedrake-install \
    --no-package-manager \
    --disable-ssh \
    --honour-petsc-dir \
    --mpicc=$MPICH_DIR/mpicc \
    --mpicxx=$MPICH_DIR/mpicxx \
    --mpif90=$MPICH_DIR/mpif90 \ 
    --mpiexec=$MPICH_DIR/mpiexec \
    --package-branch firedrake ${FIREDRAKE_BRANCH}'
    
RUN bash -c '. /home/firedrake/firedrake/bin/activate; \
   cd /home/firedrake/firedrake/src/firedrake/; \
   python -m pytest -v tests/regression/ \
   -k "poisson_strong or stokes_mini or dg_advection"'
   
RUN bash -c '. /home/firedrake/firedrake/bin/activate; \
   firedrake-clean'

This Dockerfile only has one argument: FIREDRAKE_BRANCH. This defaults to master, but you can set it to a different branch. The Dockerfile will download the firedrake-install script from this branch, which is useful if you need to modify this script because of some requirements on your target system.

The Dockerfile is invoked like so:

docker build \                                                                           
   --build-arg FIREDRAKE_BRANCH="Username/mybranch" \
   --tag=firedrake \
   --file=dockerfile.firedrake .

This creates a Docker image called firedrake in your local registry.

3. Now we convert the Docker image to a sandbox Singularity container similarly to the CI container with:

singularity build --force \
   --sandbox ./firedrake-singularity \
   docker-daemon://firedrake:latest

For the CI container we used docker://<image> to name the Docker container, which looks in the Docker-hub. For the local container we use docker-daemon://<image> which looks in the local registry.

4. Build a Singularity image from the sandbox, identically to above:

singularity build --force \
   firedrake.sif \
   ./firedrake-singularity

This creates a Singularity image called firedrake.sif that you can now copy to your target machine.