Running CDMSBats in Singularity Containers for General Users¶

These instructions were first created and tested in summer of 2022. For any further clarification about this page, contact Mithun Vanniasinghe or Yan Liu.

This document will cover CDMSBats specific instructions for installing Singularity and building and running CDMSBats for general users. Instructions for developers is the subject of future documentation.

Containers allow for a standardized environment to run CDMSBats by offering an isolated system to run CDMSBats that is reproducible and portable. Singularity is one popular container platform for creating and managing containers and the platform that we will be using to create a container for CDMSBats.

Images are immutable files that include all required dependencies to run code in a containerized environment.

In this section of the tutorial we will pull an image from the SuperCDMS container registry on gitlab and run CDMSBats in a container.

Running CDMSBats in cdmsfull_V04-07-00.sif Singularity Container¶

To run CDMSBats in a Singularity container environment on a host Linux system, we first must pull a Singularity image from the gitlab container registry. To get the image, first create a personal access token.

Next you need to pull the image from the container registry. Let’s try pulling the image for the cdmsfull image. This image comes with all required packages and makes it very convenient for general users to run CDMSBats. This image currently only has V04-07-00 of the SuperCDMS Software Release. Despite the ease of use of this image , the downside of this image is that the only version of CDMSBats that can be run in the container produced by this image is the version that comes with the image itself. Previous versions of CDMSBats can not be run and newer versions can not be run either unless the image is updated.

The important packages the image comes with includes:

IOLibrary (CDMS I/O library)
BatCommon
CDSMBats

Loading Singularity will depend on the computing site you are working on, For example, on Compute Canada, singularity can be loaded with:

module load singularity

Site specific instructions for loading Singularity are to be included in future documentation.

To pull from the registry, make sure you first ssh into your gitlab account and have access to the SuperCDMS gitlab group. Note, you will first need to create a personal access token. Then do the following commands:

singularity remote login -u<user> -p<auth_token> oras://registry.gitlab.com

singularity pull --disable-cache oras://registry.gitlab.com/supercdms/compinfrastructure/releasebuilder/cdmsfull:V04-07-00

As June 15, 2022, the latest version tag is: V04-07-00.

Now once you have pulled the image, confirm the image is in your current directory. Look for files that end in sif, (This is the Singularity Image File format). If it is, proceed to shell into the cdmsfull container with the command:

singularity shell cdmsfull_V04-07-00.sif

Once inside the container you are the same user you were on the host system. You can do the whoami command to confirm.

Singularity> whoami

Now that you have shelled into the container, you can run CDMSBats. All raw files can be accessed from the same directories as the host system. Remember to first set all required environment variables. Refer to Running CDMSBats to see a general running example.

Running CDMSBats on the cdmsbaseana_ubuntu20.04_root6.24.06.sif Singularity Container¶

Next we will try running CDMSBats on the cdmsbaseana Singularity container. This image requires everything a developer may need to write code, and commit to feature branches in the gitlab repository. Unlike the cdmsfull container, the cdmsbaseana does not come with some required packages such as BatCommon and CDMSBats. Because of this, developers are able to clone packages from the GitLab repository to use different versions of BatCommon and CDMSBats for development and testing purposes. Lets' first pull the image and then install the following packages:

IOLibrary (CDMS I/O library, required by BatCommon)
BatCommon
CDMSBats

First, let's pull the image from the container registry as before by using the command:

singularity pull --disable-cache oras://registry.gitlab.com/supercdms/compinfrastructure/releasebuilder/cdmsbaseana:<tag>

The latest tag as of June 15, 2022, is: ubuntu20.04_root6.24.06

Next, We can write a simple shell script in order to install the required packages. Here is an example procedure to write a shell script that will install the required packages:

First, create a shell script with: nano install_packages.sh

Be sure to verify that the file has execution accesses. You can check you have execution privileges with ls -l install_pacakages.sh

The three leftmost characters define permissions for the User class (i.e. the file owner). The middle three characters define permissions for the Group class (i.e. the group owning the file) The rightmost three characters define permissions for the Others class.

Let’s change read, write, and execute privileges so that only you, the file owner, has read, write, and execute privileges while the Group and Others class only has read privileges. Do this with the command:

chmod 744 install_packages.sh

Perform ls -l install_pacakages.sh once more and you should see: -rwxr--r--

Now let’s write a shell script that will clone IOLibrary, BatCommon, and CDMSBats from gitlab:

#!/bin/bash


# prompts to stop when an error is encountered
set -e

#set pointing to software directory
echo ""
echo ""
echo "This script will install IOLibrary, BatCommon, and CDMSBats if it already does not exist."
echo ""
echo "Enter directory where all software required for CDMSBats is stored or where you wish to store required software: "
read CDMS_SOFTWARE


    # installing IOLibrary
echo ""
echo ""
if [ -d $CDMS_SOFTWARE/IOLibrary ]
then
    echo "Directory to $CDMS_SOFTWARE/IOLibrary exists, nothing to be done"
    cd IOLibrary
else
    echo "Now cloning IOLibrary from GitLab"
    echo ""
    echo ""
    git clone git@gitlab.com:supercdms/DAQ/IOLibrary.git
    echo "IOLibrary successfully installed at..."
    echo $CDMS_SOFTWARE/IOLibrary
    cd IOLibrary
    make # builds the package
fi
export CDMS_IOLIBRARY=$PWD  # set environment variable
cd .. # go back one directory from IOLibrary


# installing BatCommon
echo ""
echo ""
if [ -d $CDMS_SOFTWARE/BatCommon ]
then
    echo "Directory to $CDMS_SOFTWARE/BatCommon exists, nothing to be done"
    cd BatCommon
else
    echo "Now cloning BatCommon from GitLab"
    echo ""
    echo ""
    git clone git@gitlab.com:supercdms/Reconstruction/BatCommon.git
    echo "BatCommon successfully installed at..."
    echo $CDMS_SOFTWARE/BatCommon
    cd $CDMS_SOFTWARE/BatCommon
    cmake .  # generates make files
    make -j4 install # builds the package using 4 CPU cores (this is the max number of cores on cedar)
fi
export CDMS_BATCOMMON=$PWD # set environment variables
cd .. # go back one directory from BatCommon

# installing CDMSBats
echo " "
echo " "
if [ -d $CDMS_SOFTWARE/cdmsbats ]
then
    echo "Directory to $CDMS_SOFTWARE/cdmsbats exists, nothing to be done"
    cd cdmsbats
    else
    echo "Now cloning cdmsbats from GitLab"
    echo ""
    echo ""
    git clone git@gitlab.com:supercdms/Reconstruction/cdmsbats.git
    cd $CDMS_SOFTWARE/cdmsbats
    cmake .  # generates make files
    make -j4 install  # builds the package using 4 CPU cores as this is the max on cedar
    echo "cdmsbats successfully installed at..."
    echo $CDMS_SOFTWARE/cdmsbats
fi
export CDMSBATSDIR=$PWD
cd .. # go one directory back from cdmsbats


#set environment variable to point to a required linear algebra library
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/BatCommon/BUILD/lib

We must execute the script in the container now. To execute this script we can either shell into our container:

singularity shell cdmsbaseana_ubuntu20.04_root6.24.06.sif

followed by

source ./install_packages.sh

or use the exec command:

singularity exec cdmsfull_V04-07-00.sif ./install_packages.sh

Now that we have the image and all required packages we must set the required environment variables. Additionally, we must set add the BatRoot and BatNoise to the PATH variable. This can all be done in a simple shell script called setup_and_run.sh.

Create a shell script with:

nano setup_and_run.sh

Change permissions with:

chmod 744 setup_and_run.sh

Copy the following script into setup_and_run.sh

#!/bin/sh


# Getting paths to important directories from user
while [ true ]
do
    echo "Enter path to directory where your raw data is stored: "
    read RAW_DATA_PATH
    if [ -d "$RAW_DATA_PATH"  ]
    then
        break # stop if directory exists
    else
        echo "This is not a directory, please enter an existing directory."
    fi
done

while [ true ]
do
    echo "Enter path to directory where you wish to output noise files: "
    read NOISE_FILE_PATH
    if [ -d "$NOISE_FILE_PATH"  ]
    then
       break # stop if directory exists
    else
      echo "This is not a directory, please enter an existing directory."
    fi
done

while [ true ]
do
    echo "Enter path to directory where you wish to output RQ files: "
    read RQ_FILE_PATH
    if [ -d "$RQ_FILE_PATH"  ]
    then
        break # stop if directory exists
    else
        echo "This is not a directory, please enter an existing directory."

    fi
done


while [ true ]
do
    echo "Enter path to directory where you wish to output RRQ files: "
    read RRQ_FILE_PATH
    if [ -d "$RRQ_FILE_PATH"  ]
    then
        break # stop if directory exists
    else
        echo "This is not a directory, please enter an existing directory."
    fi
done

while [ true ]
do
    echo "Enter path to directory where aux files are stored: "
    read AUX_FILE_PATH
    if [ -d "$AUX_FILE_PATH"  ]
    then
        break # stop if directory exists
    else
        echo "This is not a directory, please enter an existing directory."
    fi
done

while [ true ]
do
    echo "Enter path to directory where GPIB log files are stored: "
    read GPIB_FILE_PATH
    if [ -d "$GPIB_FILE_PATH"  ]
    then
        break # stop if directory exists
    else
        echo "This is not a directory, please enter an existing directory."

    fi
done


#env variables that can be found in tutorial
export CDMSBATSDIR=$HOME/cdmsbats
export BATNOISE_TEMPLATES=$CDMSBATSDIR/PulseTemplates
export BATROOT_PROC=$CDMSBATSDIR/UserSettings/BatRootSettings/processing
export BATROOT_CONST=$CDMSBATSDIR/UserSettings/BatRootSettings/analysis
export BATROOT_RAWDATA=$RAW_DATA_PATH
export BATROOT_NOISEFILES=$NOISE_FILE_PATH
export BATROOT_RQDATA=$RQ_FILE_PATH
export BATCALIB_RQDATA=$RQ_FILE_PATH
export BATCALIB_RRQDATA=$RRQ_FILE_PATH
export BATROOT_AUXFILES=$AUX_FILE_PATH
export BATROOT_GPIBFILES=$GPIB_FILE_PATH

#env variable that were set in install_packages.sh
export CDMS_IOLIBRARY=$HOME/IOLibrary
export CDMS_BATCOMMON=$HOME/BatCommon
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/BatCommon/BUILD/lib


#set path to executables
export PATH=$PATH:$CDMSBATSDIR/BUILD/bin

#enter commands to run BatNoise and BatRoot here:
#commands will have the following structure:
#<executable-name> -s <series number> -d <dump number> --max_events <max number of events> --processing_config <process>

#the executable name in this case would either be BatNoise or BatRoot

To execute this script we can either shell into our container:

singularity shell cdmsbaseana_ubuntu20.04_root6.24.06.sif

followed by

source ./setup_and_run.sh

or use the exec command:

singularity exec cdmsfull_V04-07-00.sif ./setup_and_run.sh

To use BatNoise or BatRoot, all there is left to do is add your commands to the setup_and_run.sh file and execute it. Refer to the Running CDMSBats tutorial for BatNoise and BatRoot commands.

Note, it is recommended to write a separate script to execute BatNoise and BatRoot commands. However, if using the exec command, BatNoise and BatRoot commands are required to be in the same shell script as the one for setting environment variables. Therefore, it is recommended to shell into the singularity container, source the setup_and_run.sh script, and source a script with CDMSBats commands instead. This will to avoid having to set environment variables each time.

Distinction Between cdmsfull and cdmsbaseana Images¶

As mentioned earlier above, the cdmsfull image is intended to be used by general users - people who just need to run CDMSBats. Since it comes with BatCommon, CDMSBats, and IOLibrary pre-installed in the image, only version tags that have been made available can be used for running CDMSBats.

The cdmsbasena image on the other hand is intended to be used by developers. Since the image does not have BatCommon, CDMSBats, or IOLibrary pre-installed, the user must clone the repo themselves. This allows them to access the previous and latest versions of CDMSBats for development and testing purposes.