Production setup at remote sites

There is a description of the prod/ tree in the production area of the simu web page. You need to replicate needed parts of this structure under $BFROOT/prod (which can of course be a link to wherever you want). You do not need to copy over everything in this tree.

To obtain what you need, first check out the SimProd CVS module while you are in the prod directory to generate prod/SimProd. Check it out from SLAC with 'cvs co -r VXX-XX-XX SimProd' where VXX-XX-XX is the version you need. (I am assuming CVS is set up properly.) Then issue 'gmake' in SimProd to build the other important part of the prod tree, the prod/mgr directory.

prod/mgr contains scripts to build and run production jobs at your site. All but the local-* scripts are copied from prod/SimProd to prod/mgr by SimProd's gmake procedure. If the files already exist in prod/mgr, they are renamed to filename.old. These files normally are not modified by the local site; hence the arbitrary disposal of any existing local copies.

The local-* files are treated in a special way. If copies already exist in prod/mgr they are not touched. If they do not exist, they are copied from prod/SimProd. This is because these files are modified locally, and existing local copies must be respected. The purpose of these files is described below. The important files installed in prod/mgr by SimProd's gmake:

• defineRuns. The master list of run definitions used to build individual job directories. The local site uncomments the run blocks it has been allocated, and runs defineRuns as a script to generate the job directories in prod/log/allruns.
• buildRuns. The script called by defineRuns to build the job directories and files. Doesn't need to be touched.
• local-batch. The contents of this file is inserted at the top of each job script built by defineRuns/buildRuns. Many batch systems accept batch directives at the top of batch scripts; this file is used to insert any such directives needed. For example, this file at Colorado looks like

  #$ -s /bin/sh
  #$ -eo /home/axp/hpark/batch.log
  #$ -mu hpark@pizero.colorado.edu
  #$ -mb
  #$ -me
  #
  

  Note that if you modify this file, you have to rerun defineRuns to generate new job files incorporating the modification.
• local-setup. This script is called before BBSIM is started to set up the job's output area and do any other preparatory work. See the distributed example for how to use it. Its default behaviour is to create an output directory named with the run number and send all output files to this directory. You should preserve this behavior; this is our standard for organizing production output.
• local-wrapup. This script is called after the job has completed. It can be used, for example, to tar the run directory and move it to tape.
• bbprod. This is the script that actually runs BBSIM. It is called in the job scripts generated by defineRuns/buildRuns. It normally doesn't need to be touched. Note that it writes job status files to prod/log/status/ to enable monitoring of job execution.

Then from prod/mgr run defineRuns once you've uncommented the runs you intend to process. This script contains the specification of how runs are defined. A combination of generator data file, FFREAD input file, and database input file are passed to the buildRuns script to define a group of runs, eg.

./buildRuns RUN1=10261 NRUN=112 DATF=background/heb_960120/heb_960120.dat. \
   DAT1=1 NDAT=1 SRT=0.6.3 FFR=requests/trg/ray.ffr  DB=requests/trg/nominal.db

Note that buildRuns is a perl script; it expects to find Perl in /usr/local/bin/perl. Perl is on the BaBar 'required software' list and can be obtained via ftp at prep.ai.mit.edu:/pub/gnu/.

Running defineRuns will generate the run job files in prod/log/allruns. The files for each run (.ffr, .db, .list, .job) are stored in individual run directories named with the run number. Note the .job files; these are the actual production jobs. They call the script bbprod which actually does the run. You will see in the .job files that a local setup script is called, prod/mgr/local-setup, so you can control (via options passed to bbprod in the LOCALPRODOPTS env variable, or environment variables directly interpreted by the bbprod script; see 'bbprod help') what directory the job runs from, where output is written, etc. For example, the Colorado local-setup file looks like

#!/bin/sh
#
# Use this script to set up local preferences in the $LOCALPRODOPTS
# environment variable which is passed to bbprod (see 'bbprod help'
# for options).
# Run number and SRT version come into this script as parameters
# 1 and 2. If that doesn't work, they also come in on the JOBNUM
# (run number) and JOBSRT (SRT version) environment variables.
# This script is sourced ('. local-setup') from the job script so
# the environment it sets up is propagated back to the job.
#
# send output and log material all to the same directory,
# a run-specific directory where the job also runs from
# (specified in the $FROMDIR parameter; bbprod creates the
# directory).

BFROOT=/axp/babar/TestDir; export BFROOT
BFDIST=${BFROOT}/dist; export BFDIST
BB_LOST=$BFROOT/test/lostpcl
LOGDIR=$BB_LOST/log/out/$JOBNUM ; export LOGDIR
FROMDIR=$LOGDIR ; export FROMDIR
OUTDIR=$LOGDIR ; export OUTDIR

And from the bbprod help listing:

Usage: bbprod [options]
  Options (with default):
        -srt srtno             (SRTNO=)
            SRT release      (LOGDIR=/axp/babar/TestDir/prod/log/out)
            Directory where log material goes
        -exe dir               (EXEDIR=/nfs/juno/u5/prod/bin/AIX2)
            Directory in which bbsim, gnbase programs are looked for
        -run runno             (RUNNO=000000)
            Run number; must be 6-char w/leading 0's
        -from dir              (FROMDIR=./)
            Directory from which job is run. It is created if it doesn't exist.
        -in dir                (INDIR=/nfs/juno/u5/prod/log/allruns)
            Directory in which input files are looked for
        -out dir               (OUTDIR=./)
            Directory where output files are directed
        -dat dir               (DATDIR=/nfs/juno/u5/dist/releases//gnbase/dat)
            Directory where bbsim data files (eg. field map) are looked for
        -log logdir            (LOGDIR=/nfs/juno/u5/prod/log/out)
            Directory where log material goes
        -nice n                
            Use 'nice -n pgmname ...'
        -trace    
            Turns script echoing on
Defaults can be overridden either using the appropriate command line
option or by setting the environment variable indicated.
'bbrun help' to get this help listing.

With no options, bbprod will run without a 'cd' anywhere and will write its output to the local directory. There is also an end of job script, prod/mgr/local-wrapup eg. to move output files to a final location.

bbprod will look for the bbsim and gnbase executables in

      $BFROOT/prod/bin/$BFARCH/bbsimbVV.exe
      $BFROOT/prod/bin/$BFARCH/gnbabarVV

      $BFROOT/prod/bin/$BFARCH/bbsimb1.1.0.exe
      $BFROOT/prod/bin/$BFARCH/gnbabar1.1.0

Once you have configured your local-setup, local-wrapup files as you like, you can start submitting the .job files allocated to you as batch jobs. Note that the job scripts assume no information from the environment; everything they need to know they define for themselves.