General Running Example¶

Running CDMSBats requires the specification of various arguments, it helps to quickly go over these arguments before we jump right into attempting to run these programs. A basic command has the following structure:

  <executable-name> -s <series number> -d <dump number> --max_events <max number of events> --processing_config <processing_file_name> --analysis_config <analysis_file_name>

The first argument we must establish is the executable name. That is, we must specify which CDMSBats program we would like to use, i.e. BatNoise or BatRoot.

Afterwards comes the series number, this is a unique number that is used to identify the type of data we are running.

Next is the dump number; within a series there are multiple dumps. Dumps are varying input files that are run on the same settings as the rest of the series.

Then comes the max events argument. For BatNoise, this is the number of maximum events to be processed in total. Whereas for BatRoot, this is the number of events per dump.

Lastly, there are the processing and configuration (or analysis) files. When running CDMSBats programs, we should always define which processing and configuration files to use (Note, you do not have to specify processing and configuration files and default files will be used, but this is not good practice).

Usually, we find that with a particular series there are corresponding processing and configuration files we should use. Start August 2022, a list of these files can be found at a separate repository (cdmsbats_config): /somedirectory/cdmsbats_config/UserSettings/BatRootSettings/analysis or /somedirectory/cdmsbats_config/UserSettings/BatRootSettings/processing.

We can turn on and off the analysis of different detectors and analysis algorithms from within the processing file. The configuration file contains parameters used by BatRoot for a particular run of data as well as cut parameters used by BatNoise. Collectively, the processing and configuration files tell CDMSBats how to process raw data by designating what parameters and algorithms to use.

To successfully use the BatNoise and BatRoot programs, we must have the correct repositories cloned from GitLab; these repositories include IOLibrary, BatCommon and CDMSBats.

First, we should emphasize the importance of setting the correct environment variables, this is vital for not only running CDMSBats but also for recompiling. The following is a list of environment variables:

CDMSBATSDIR - Directory where you checked out "cdmsbats". Ex: "/somedirectory/cdmsbats"
BATROOT_CONST - Path to analysis files, located outside cdmsbats. Ex: "/somedirectory/cdmsbats_config/UserSettings/BatRootSettings/analysis"
BATROOT_PROC - Path to processing files, located outside cdmsbats. Ex: "/somedirectory/cdmsbats_config/UserSettings/BatRootSettings/processing"
BATCALIB_CONST - Path to calibrartion files, located outside cdmsbats. Ex: "/somedirectory/cdmsbats_config/UserSettings/BatCalibSettings/calibration"
BATCALIB_PROC - Path to processing files, located outside cdmsbats. Ex: "/somedirectory/cdmsbats_config/UserSettings/BatCalibSettings/processing"
BATNOISE_TEMPLATES - Path to pulse templates, located outside cdmsbats. Required by BatNoise and BatRoot. Ex: "/somedirectory/cdmsbats_config/PulseTemplates/"
BATROOT_RQDATA - Path to where BatRoot output files (.root files) will be placed. Will be used by BatCalib later.
BATCALIB_RRQDATA - Path to BatCalib output files (.root files).
BATROOT_NOISEFILES - Path to where the noise files generated by BatNoise (.root files).
BATROOT_RAWDATA - Path to where raw data files (.gz file types).
BATROOT_GPIBFILES - Path to GPIB files*.
BATROOT_AUXFILES - Path to all other auxilliary files (.dmm, .info, .isr)*.

(The last two are exclusive to using CDMSBats for Soudan files)

On Confluence, there are multiple resources and examples of how you can set up environment variables and compile CDMSBats depending on the platform you are working on. These resources can be found here.

(In the following section we will set environment variables as if you are working on Cedar, make sure you check these resources above)

EXAMPLE¶

For the most part, executing BatNoise and BatRoot are very similar. But, let us start with going into how to run BatNoise. As mentioned before, BatNoise is the first step in the overall toolchain. BatNoise should be run first because it produces a filter file which is used by BatRoot later on in the optimal filter algorithm.

First, let's go ahead and create directories to hold our raw data and the intermediary files from each of the tools we'll be using to process that data. Ideally, we want individual directories to store raw data, noise files, RQ data, RRQ data, any auxiliary files, and GPIB log files.

  (base) [loki@16f0305f3528 ~]$ mkdir -p raw
  (base) [loki@16f0305f3528 ~]$ mkdir -p noise
  (base) [loki@16f0305f3528 ~]$ mkdir -p rq
  (base) [loki@16f0305f3528 ~]$ mkdir -p rrq
  (base) [loki@16f0305f3528 ~]$ mkdir -p aux
  (base) [loki@16f0305f3528 ~]$ mkdir -p gpib

Each of BatNoise, BatRoot, and BatCalib will be looking for files in their relevant folders, specified by the environment variables. Here is a copy-pastable chunk of text that will let you set all of them pointing to the correct directories, including the directories we just created. (Starting August 2022, note that you have to clone the cdmsbats_config repository on the same directory level as cdmsbats repository)

  [ -n "$CDMSBATSDIR"] || export CDMSBATSDIR=$HOME/cdmsbats
  export BATNOISE_TEMPLATES=$CDMSBATSDIR/PulseTemplates
  export BATROOT_PROC=$CDMSBATSDIR/../cdmsbats_config/UserSettings/BatRootSettings/processing
  export BATROOT_CONST=$CDMSBATSDIR/../cdmsbats_config/UserSettings/BatRootSettings/analysis

  export BATROOT_RAWDATA=$HOME/raw
  export BATROOT_NOISEFILES=$HOME/noise
  export BATROOT_RQDATA=$HOME/rq
  export BATCALIB_RQDATA=$HOME/rq
  export BATCALIB_RRQDATA=$HOME/rrq
  export BATROOT_AUXFILES=$HOME/aux
  export BATROOT_GPIBFILES=$HOME/gpib

Go ahead and take a look at what environment variables are set. You will see something along the lines of

  (base) [loki@16f0305f3528 ~]$ env | grep -e "BAT" | sort
  BATCALIB_RQDATA=/home/loki/rq
  BATCALIB_RRQDATA=/home/loki/rrq
  BATNOISE_TEMPLATES=/home/loki/cdmsbats/PulseTemplates
  BATROOT_AUXFILES=/home/loki/aux
  BATROOT_CONST=/home/loki/cdmsbats/UserSettings/BatRootSettings/analysis
  BATROOT_GPIBFILES=/home/loki/gpib
  BATROOT_NOISEFILES=/home/loki/noise
  BATROOT_PROC=/home/loki/cdmsbats/UserSettings/BatRootSettings/processing
  BATROOT_RAWDATA=/home/loki/raw
  BATROOT_RQDATA=/home/loki/rq
  CDMSBATSDIR=/home/loki/cdmsbats

You might also wish to add the exports for the environment variables to your ~/.bashrc, that way these variables will be set every time you start and run the container. Just remember that you set them there!

Accessing Data¶

At this point, we will be working with some real reference data from CUTE, available from here:

https://srs.slac.stanford.edu/DataCatalog/folder.jsp?folder=42257403

For a full set of instructions on downloading data visit the Downloading Raw Data page. In the following example, wget will be used to download raw data.

We are interested in the first data dump, F0001. Go ahead and switch into our raw directory with cd raw, and then let's use wget to download that data file.

  (base) [loki@16f0305f3528]$ wget https://srs.slac.stanford.edu/DataCatalog/get?datasetLocation=42291893 -O $HOME/raw/23190928_1623_F0001.mid.gz

It might take a little while over a small connection, it's a lot of data!

Please note that the data follows a specific naming convention. Here, 23190928_1623 is the series, and _F0001 is the dump number.

BatNoise¶

Now, we can begin to process. BatNoise performs a Fast-Fourier Transform to identify "Noise" frequencies and identify "cuts" of frequencies of interest from different events. It produces a NoiseFilter file, which will be used by BatRoot. As with most programs, you can run BatNoise without arguments and a usage dialogue will appear.

Note that we do not need to be in the directory containing the data, as we have specified using environment variables where the raw data is and where we will write the "noise" file.

Here is an example command which will use a specific analysis and detector configuration options file meant for CUTE encoded in the Midas format.

For an explanation of arguments, run BatNoise --help. Run it!

$ BatNoise -s 23190928_1623 -d 1 --max_events 5000 \
      --processing_config processingCUTEData.G124 --analysis_config configCUTEData.G124

As it is running, it will print some diagnostics and what it has found to the console session. Once it has been completed, you should be able to use ls inside our chosen output directory, noise/.

  (base) [loki@16f0305f3528 noise]$ ls -alh
  total 129M
  drwxr-xr-x 2 loki root 4.0K Oct 17 06:52 .
  drwx------ 1 loki root 4.0K Oct 17 06:29 ..
  -rw-r--r-- 1 loki root 109M Oct  1 07:25 23190928_1623_F0001.mid.gz
  -rw-r--r-- 1 loki root  21M Oct 17 06:52 Test_Filter_23190928_1623.root

As the program output lets us know, it has written out the cut distributions to Test_Filter_23190928_1623.root.

Another useful accessory of the BatNoise program is that we can process multiple dumps at a time and write the output to a single filter file. For example, if we wanted to run BatNoise for dumps 1 through 25 (on the same file as above), we can write the command:

 $ BatNoise -s 23190928_1623 -d 1-25 --max_events 5000 \
      --processing_config processingCUTEData.G124 --analysis_config configCUTEData.G124

We should also note that there are specific processing, analysis, and calibration files available for data from different sources. There are default configuration files available from the repository, so we potentially could have run

  $ BatNoise -s 23190928_1623 -d 1 --max_events 5000

BatRoot¶

BatRoot is the next step in the toolchain. BatRoot will use the filter file we created with BatNoise to generate the Reduced Quantities which will be useful for the interpretation of the data.

It will use the same analysis and processing files that BatNoise did. Let's run it.

  $ BatRoot -s 23190928_1623 -d 1 --max_events 5000 \
        --processing_config processingCUTEData.G124 --analysis_config configCUTEData.G124

After it's done, let's check our rq directory.

  (base) [loki@16f0305f3528 rq]$ ls -alhtr --color
  total 130M
  -rw-r--r-- 1 loki root 109M Oct  1 07:25 23190928_1623_F0001.mid.gz
  drwx------ 1 loki root 4.0K Oct 17 06:29 ..
  -rw-r--r-- 1 loki root  21M Oct 17 07:29 Test_Filter_23190928_1623.root
  drwxr-xr-x 2 loki root 4.0K Oct 17 07:29 .
  -rw-r--r-- 1 loki root 464K Oct 17 07:30 Test_23190928_1623_F0001.root

Now, we can see a Test_23190928_1623_F0001.root file has been written out. Yay!

BatRoot can process only one dump at a time and it will write the output of each dump to a separate root file. To obtain a single output file for multiple dumps from BatRoot, you have to merge the root files after running BatRoot. You can either write a script to do this or the easier way is to use the "hadd" functionality of root that allows you to easily merge root files. The basic command is:

  hadd <merged_file_name> <input file1> <input file2>

This command will merge input file1 and input file2 into one file with the merged_file_name. An example is shown below. When you have a large number of files like below, the part with {1..9}, {10..99} etc will automatically be expanded in the range (1-9), (10-99) etc to include all files with those names. For Example:

  Test_23190928_1623_F000{1..9}.root

is equivalent to:

  Test_23190928_1623_F0001.root
  Test_23190928_1623_F0002.root 
  ...
  Test_23190928_1623_F0009.root.

To merge the data set from beforehand, we can use the command:

  $ ROOTSYS/bin/hadd Test_23190928_1623.root Test_23190928_1623_F000{1..9}.root Test_23190928_1623_F00{10..99}.root Test_23190928_1623_F0{100..999}.root Test_23190928_1623_F{1000..2375}.root