Validation for Monte Carlo Production

This note is an attempt at a procedure for validating sites for BaBar Monte Carlo production. Since the code itself is not stable these can be taken as guidelines not as "Authoritative Recipes".

A. Setting up your systems and Environment

1. Required Software

   Check the Required Software for BABAR Computing page for the software that will need to be installed at your site.

2. BFROOT

   Your site should be properly set up with a BFROOT tree that has at least the dist subdirectory tree and user environments with BFROOT, BFDIST, and BFSITE defined.

3. Production Directories

   Two extra subdirectory trees are needed for production running: prod and releases. You can create this by running:

   > mkdir -p $BFROOT/releases
   > mkdir -p $BFROOT/prod/log/allruns
          

   The first will hold the lettered releases which can be in the 1 GB range. The second will hold the application run directories. This can get large too with log and hbook files taking a few MB per run. Also, simulation XDR files can be upto 200MB per run.

4. Production Account (optional)

   At SLAC we have defined the group account bbrprod for running all production jobs. This removes the hassle of dealing with directory/file/batch permissions for multiple users.

B. Getting the Software

1. Know what releases you need.

   The full information is documented in the HOWTO package in HOWTO-Setting-Up-SP3-FDB. This also documents most other aspects of production.

2. Get and install the necessary base production release(s).

   This step is done using the SRT utilities importrel and importarch. For example:

   > cd $BFDIST/releases
   > importrel -p -a 7.11.2
   > importrel -p -a 7.12.7
   > importrel -p -a 8.1.10
       

   (each takes a fair amount of time - checks for existence of packages making up the release, and loads those versions not already present.)

   > importarch -p 7.11.2 OSF1V4 
   > importarch -p 7.12.7 OSF1V4-Optimize
   > importarch -p 8.1.10 OSF1V4-Optimize
       

   Of course, change OSF1V4 above to SunOS5 if you use Solaris. This step brings over the HUGE libraries and binaries. I am guessing (but have not verified) that you do not need the libraries which are the biggest part of this import. To save time and space, you may wish to just copy over some of the binaries you may need (like BdbFileConfigLoader) for creating and/or setting up the database. Sorry, I don't have a comprehensive list of these so if someone does, please email me.

3. Get and install the necessary lettered production release(s).

   Some earlier production releases can be found tarred and compressed at $BFROOT/prod/dist. This is where 7.11.2f and 7.12.7b can be found. More current production release can be found in $BFROOT/releases/. Production releases are documented on their own web page.

   Below is a simple procedure for importing a lettered release to your site. For example, for 8.1.10h for Solaris, use:

   > mkdir $BFROOT/releases/8.1.10h
   > ssh flora gtar cf - -C /afs/slac/g/babar/releases/8.1.10h \
        --exclude OSF1V4-Optimize . | gtar xvf - -C $BFROOT/releases/8.1.10h
   > cd $BFROOT/releases/8.1.10h
   > rm BASE
   > ln -s $BFDIST/releases/8.1.10 BASE
   > cd workdir
   > gmake setup
       

   Notes:

   • instead of the mkdir, you may which to make symlink to where you have space. 8.1.10h takes over 800MB for both architectures.
   • additional --exclude options can be give to exclude other archs (such as Linux) when they appear. Also both plain and optimized archs may need excluding. Check the bin directory of the release at SLAC.
   • If they are any Elf* binaries, feel free to delete them. You can also use the --exclude option on these.

   IMPORTANT: After the import is done, you will need to cd to the workdir directory of the release and run 'gmake setup'.

4. Create/setup the the federated database for Production

   Both the SimApp and Bear stages (mixing and reconstruction) utilize the Objectivity database. Install and configure the database in a manner consistent with the SLAC installation.

   Further info is in HOWTO-Setting-Up-SP3-FDB.

5. Get, install and configure the ProdTools package in $BFROOT/prod.

   Get the latest version of the ProdTools package. The easiest way to do this is to use CVS if you have access to the SLAC repository from your site. For example:

   > cd $BFROOT/prod
   > cvs co ProdTools
        

   If you have checked it out previously, cd into the ProdTools directory and do a cvs update -Ad.

   You will need to write a site specific portion of ProdTools to use it. There is some sketchy information on this in the README and in the comments. Every site needs to create a subdirectory under ProdTools/site with a name equal to the setting of $BFSITE. Use the contents of one of the other remote sites as a template for your site as the SLAC ones are almost certainly too complex for most. The ProdTools package is coordinated by Douglas Smith.

6. Get the background, configuration and conditions - see HOWTO-Setting-Up-SP3-FDB.

C. Running Production

Once you have installed at least the simulation release you can start running. The objective is to do several runs and compare them to results at SLAC. The comparisons are based on the log files, the histograms and the database output.

Running a production job is a 3 stage effort. Each stage is a 4 to 6 step operation. The three stages are simulation, which produces an xdr file. mixing and digitization, which produces a database collection and reconstruction that reads the digitized database and produces a different database collection.

The three stages are distinguished by the -t flag to the ProdTools commands: -t simu, -t mixr and -t reco. The five steps to run each level of production job are identified by the five scripts in ProdTools run for that step:

1. Simulation Stage

   To define and run the first 3 simulation final states:

   > spbuild -t simu --nodb RELEASE=7.11.2f \
     DECFILE='B0B0bar_JpsiKS_+-.dec' GENERATE='Upsilon(4S)' NEVENTS=1000 \
     DESC='B0 -> X, B0~ -> J/psi(e+e-/mu+mu-)K0s(pi+pi-) + c.c' QAREF=01010 091000-091001
   
   > spbuild -t simu --nodb RELEASE=7.11.2f \
     DECFILE='B0B0bar_Dstrlnu.dec' GENERATE='Upsilon(4S)' NEVENTS=1000 \
     DESC='B0 -> X, B0~ -> D*(2010)+ l- nub + c.c.' QAREF=01020 091002-091003
   
   > spbuild -t simu --nodb RELEASE=7.11.2f \
     DECFILE='B0B0bar_JpsiKS_00.dec' GENERATE='Upsilon(4S)' NEVENTS=1000 \
     DESC='B0 -> X, B0~ -> J/psi(e+e-/mu+mu-)K0s(pi0pi0) + c.c' QAREF=01010 091004-091005
   
   > spsub --nodb -t simu -r 7.11.2f 091000-091005
   

   The "next" option makes life even simpler - you can have the simu job submit mixr and the mixr job submit reco upon completion. This has a number of advantages including simplicity, ensuring a mix of jobs, and ensuring that needed files are not staged to tape and deleted from disk before the next stage of the job can run. The spsub command is modified:

   > spsub -y --nodb -next reco -t simu -r 7.11.2f 091000-091005
   

   Note the use of "next" and the "y" option to automatically submit the job.

   While the job is running you can run:

   > spupdate --nodb -t simu -r 7.11.2f 091000-091005
   

   This will give you the status of the job if the log files are reachable from the machine you run it from.

   Then you can cleanup and filter the log file using:

   > spclean --nodb -t simu -r 7.11.2f 091000-091005
   

   More details and examples can be found in the ProdTools/README file. The decay files used to define the simulation jobs are found in the ProdDecayFiles package that comes with the lettered simulation release.

2. Mixing/Digi Stage with background

   > spbuild --nodb -t mixr -B x1 RELEASE=8.1.10h \
     PROD_FD_BOOT=$OO_FD_BOOT 091000-091005
   
   > spsub --nodb -t mixr -B x1 -r 8.1.10h 091000-091005
   
   > spupdate --nodb -t mixr -B x1 -r 8.1.10h --dofail 091000-091005
   
   > spclean --nodb -t mixr -B x1 -r 8.1.10h 091000-091005
   

3. Reconstruction Stage with background

   > spbuild --nodb -t reco -B x1 RELEASE=8.1.10h \
     PROD_FD_BOOT=$OO_FD_BOOT 091000-091005
   
   > spsub --nodb -t reco -B x1 -r 8.1.10h 091000-091005
   
   > spupdate --nodb -t reco -B x1 -r 8.1.10h --dofail 091000-091005
   
   > spclean --nodb -t reco -B x1 -r 8.1.10h 091000-091005
   

4. Mixing/Digi Stage without background

   > spbuild --nodb -t mixr -B x0 RELEASE=8.1.10h \
     PROD_FD_BOOT=$OO_FD_BOOT 091000-091005
   
   > spsub --nodb -t mixr -B x0 -r 8.1.10h 091000-091005
   
   > spupdate --nodb -t mixr -B x0 -r 8.1.10h --dofail 091000-091005
   
   > spclean --nodb -t mixr -B x0 -r 8.1.10h 091000-091005
   

5. Reconstruction Stage without background

   > spbuild --nodb -t reco -B x0 RELEASE=8.1.10h \
     PROD_FD_BOOT=$OO_FD_BOOT 091000-091005
   
   > spsub --nodb -t reco -B x0 -r 8.1.10h 091000-091005
   
   > spupdate --nodb -t reco -B x0 -r 8.1.10h --dofail 091000-091005
   
   > spclean --nodb -t reco -B x0 -r 8.1.10h 091000-091005
   

D. Validation

Once some number of jobs have been run we would like to compare them to the comparable files at SLAC. There are a set of comparison tools developed by Artem Trunov (artem@slac.stanford.edu) to compare log files and histogram files for each of the runs. These tools are similar to the QA tools that compare various runs with a standard set. To do the comparison the log files and histogram files must be moved to SLAC. Transfer the *log.gz, framework.hbook, and qasave.rzn files to your site's directory under /afs/slac.stanford.edu/g/babar/prod/log/validation. Then do the comparison following the instructions.

Data Export is documented here.

DataBase Configuration for Monte Carlo Production is covered here.