Beta/Kanga Tutorial -- May 15, 2000

This tutorial demonstrates how to run a very simple Beta job in two modes:

reading data from the Objectivity event store and writing histograms to an Hbook file (Exercise 1); and
reading data from a Kanga file and writing histograms to a ROOT file (Exercise 2).

and then, how to run a longer, more complicated job in batch (Exercise 3). The output of this last job can be used with ROOT. See the ROOT tutorial documentation for more information.

Note: This tutorial is most likely to be helpful to you if you have not run any Beta job in many months (or at all). If you have run Beta successfully in the last two or three months, you probably already know most of what's here.

This tutorial is created partially from material found in the BaBar Offline Workbook, the Kanga Home Page, and the Physics Analysis Recipes.

To execute this tutorial successfully, you will need:

a unix account properly configured for the BaBar environment (i.e., one which executes the hepix scripts at login);
10 MB free space on disk;
scratch disk space for your account on $BFROOT/work (this can be easily created if you have never used it before).

Exercise 1. Reading from Objy and writing an Hbook file

In Exercise 1 we will:

prepare a test release for a simple Beta job that reads event data from an Objectivity federation;
compile the code;
link the code;
run the job; and
look at the resulting histograms with paw.

Logon to a Unix platform. At SLAC this is usually shire or tersk.
We will be using the first of a new series of releases of BaBar software specifically designed for people doing physics analysis. It will incorporate only those changes that affect physics and will therefore tend to be more stable than the production releases.
To initialize the software release tools for this release, called 8.6.2d-physics-1, and your platform, type:
```
   > srtpath 8.6.2d-physics-1
   Select/enter BFARCH (CR=1):
   1) SunOS5
   2) SunOS5-noOptimize-Debug
```
(Text in brown indicates output from the shell or other program. Also, a carriage-return is always implied at the end of each line of user input.)
Choose SunOS5 (or OSF1V4 or Linux2, depending on your platform) by typing "1" and hitting return.
The srtpath command modifies your unix path so that any files you need while you work with BaBar code will obtained from the right place. Different files may be required for different architectures (Sun, Linux, OSF) or different configurations of the software (versions with or without debugging information, for example). We have chosen the version of the code that is optimized and contains no debug info, for speed and compactness.
Select a directory to contain your test releases. (A test release is a directory tree on disk that will contain the necessary subdirectories to edit, compile, link, and run a BaBar analysis job.) Choose a location with at least 10 MB free space. We have chosen the directory mybabar, but you may choose any other, even your home directory. cd to this directory now.
```
   > cd mybabar
```
If you haven't already, create your scratch directory, $BFROOT/work/<first_letter>/<your userid>. For example, if your userid was rfc:
```
   > mkdir $BFROOT/work/r/rfc
```
Create your test release directory using the newrel command. Use the option
```
   -s $BFROOT/work/<first_letter>/<userid>
```
to specify your scratch area. For example, user rfc would type:
```
   mybabar> newrel  -s $BFROOT/work/r/rfc -t 8.6.2d-physics-1 Beta_tt
```
The last argument provides the (arbitrary) name for the new directory that will contain the test release.
(The unix prompt will be indicated by ">" unless the command must be issued from a particular directory, in which case, the deepest subdirectory will appear, in this case, "mybabar>.")
It is possible to incorporate several executable versions of the code in the same test release, for different architectures, for example. For more information on this click here.
Switch to your new test release directory:
```
   mybabar> cd Beta_tt
```

Now we add the packages we will need:

   Beta_tt> addpkg workdir
   Beta_tt> addpkg BetaUser

Next, select the Objectivity federation where the data you want to use resides. To use recent BaBar data, you would ordinarily type:
```
  Beta_tt> analboot2
```
But, if the production federations are blocked, as they would ordinarily be during the hour of this tutorial, type instead:
```
   Beta_tt> setenv OO_FD_BOOT /nfs/objyserv6/objy/databases/test/physics-demo/V1/BaBar.BOOT
```

Switch to the BetaUser directory and copy in a simple example from the BaBar offline workbook, then switch back to your test release:
```
   Beta_tt> cd BetaUser
   BetaUser> cp $BFROOT/www/doc/workbook/examples/NTrkExample/* .
   BetaUser> cd ..
```
(Note: the "." at the end of the second line is part of the copy command!)
Now we can compile and link. Always use batch for this and always do this from your test release top-level directory and always use the bldrecoq queue:
```
   Beta_tt> bsub -q bldrecoq -o Exercise1.log gmake lib ROPT=-noDebug-Shared
```
(The procedure at non-SLAC sites may be different; check with your local guru.)
"-noDebug" turns off debugging symbol generation, for speed and code size, and "-Shared" allows you to use shared libraries, which means that if you modify your own code, you need only re-compile it, and re-linking is not necessary. The ROPT variable may be set to achieve the same effect:
```
   > setenv ROPT -noDebug-Shared
```
We can monitor the progress of the batch job with:
```
   > bjobs      (use option -l for more info)
```
(Text in green is informational; you don't type it.)

When the compile has finished successfully, submit the link job:

   Beta_tt> bsub -q bldrecoq -o Exercise1.log gmake BetaUser.bin ROPT=-noDebug-Shared

When the link has completed, switch to the workdir from which the job will run:
```
   Beta_tt> cd workdir
```
Now prepare the workdir directory for use:
```
   workdir> gmake setup
```
This need be done only once, before the first time you run from it.
We have chosen to analyze run 12125. We need to determine the collection name for the event collection containing the reconstructed physics data for this run. There are several ways to do this. We are going to prepare a .tcl file containing the necessary instructions. To do this for run 12125, type:
```
   workdir> getdata -t allevents -b12125 -e12125 -v
```
This will produce a file called allevents_12125-12125.tcl, containing the tcl commands which can be source'd at run time to choose your file(s). To examine the contents of this file, type:
```
   workdir> cat allevents_12125-12125.tcl

   # allevents collections from analboot2
   mod talk BdbEventInput
     input add /groups/isPhysicsEvents/0001/2100/P8.6.1aV00fb/00012125/cb001/allevents
   exit
   # No. of allevents events is 50564 for runs 12125-12125
   # The production federation for these collections is analboot2
```
If the production database is blocked, we instead do the following:
```
   Beta_tt> setenv BdbInputCollection /groups/isMultiHadron/0001/2100/P8.6.1aV00fb/00012125/cb001/allevents
```
(If you'd like more information on finding and setting input collections, click here.)
We're ready to go. We start the run by invoking the program betarun with the name of the relevant .tcl file as an argument:
```
   workdir> betarun ../BetaUser/bdbMicro.tcl
```
If we are using the special private federation, you have already specified the correct collection by setting the BdbInputCollection variable. Otherwise, do this now by typing:
```
   > source allevents_12125-12125.tcl
```
Now process three events:
```
   > ev beg -nev 3
```
Now let's continue with the next 47 events for a total of 50:
```
   > ev continue -nev 47
```
When it finishes and leaves you at the tcl prompt again, type:
```
   > exit
```
We can now look at the Hbook output. Enter:
```
   workdir> pawX11
   ...
```
Hit a carriage return to select the default Xwindows output.

Open the Hbook file and display its contents:

   PAW > h/fil 1 framework.hbook
   PAW > h/lis

Set some optional statistics display parameters and then display the momentum histogram:
```
   PAW > opt stat
   PAW > set stat 111111
   PAW > h/pl 1
```
Exercise 1 is complete. Exit from paw:
```
   PAW > exit
```

Exercise 2. Reading from Kanga and writing a ROOT ntuple file

In Exercise 2 we will:

prepare a test release for a simple Beta job that reads event data from a Kanga file;
compile the code;
link the code;
run the job; and
look at the resulting histograms with ROOT.

A bit of introduction:

Kanga (a Kinder ANd Gentler Analysis) provides files containing the same information as that in the Objectivity event store and conditions database, but without the complexities of Objectivity. User code is exactly the same in each case, but the executable must be remade for Kanga. (It's also possible to make a dual-purpose executable that can be used for both types of data, depending on the settings of some environment variables.) The advantages of using Kanga are increased throughput and access to data when the production databases are blocked.

Kanga files can be used in any analysis that depends only on variables contained in the micro-dst. If you need to study hits in the drift chamber, for example, you must get your events from Objectivity raw data.

ROOT is a histogram and analysis package from CERN, and is supposed to be a modern replacement for HBook/PAW (although HBook files can be converted to be used in ROOT). The data in Kanga files are stored in ROOT trees (similar to HBook ntuples), but reading Kanga files doesn't commit you to using ROOT to make your histograms. You can read Kanga files and produce HBook output, and you can read Objectivity files and produce ROOT output.

Let's begin:

Go back to the directory you selected in Exercise 1 to contain that test release (the one where you did the first newrel command). For example:
```
 > cd ~/mybabar
```
We assume the software release tools are still set up for your platform and release 8.6.2d-physics-1. If not (for instance, you have logged out and back in since doing Exercise 1), initialize the software release tools for release 8.6.2d-physics-1 and your platform again:
```
   mybabar> srtpath 8.6.2d-physics-1
```
Choose SunOS5 (or OSF1V4 or Linux2) by typing "1" and hitting return.
Create a new test release directory and cd to it. (Remember to replace r/rfc in the example below with your userid and first letter.)
```
   mybabar> newrel -s $BFROOT/work/r/rfc -t 8.6.2d-physics-1 Kanga_tt 
   mybabar> cd Kanga_tt
```

Add packages and initialize workdir:

   Kanga_tt> addpkg BetaUser
   Kanga_tt> addpkg workdir
   Kanga_tt> gmake workdir.setup

Now set some environment variables. (Again, don't enter the text in green--it just explains the purpose of each variable.)

   > setenv BetaKanga         yes  (allows reading from Kanga File)

   > setenv BetaBdbMicro      no   (disables reading from the database)
   > setenv BetaRootTuple     yes  (make a ROOT, rather than an Hbook, file)

As before, for convenience and speed:

   > setenv ROPT -Shared-noDebug

Copy in the example from the workbook:

   Kanga_tt> cp $BFROOT/www/doc/workbook/examples/NTrkExample/* BetaUser/

Compile and link. Since we know the code will compile and link, we can combine the two steps and save a bit of time:
```
   Kanga_tt> bsub -q bldrecoq -o Exercise2.log gmake lib BetaUser.bin
```
(The ROPT option has been taken care of in by setting the environment variable.)
Make the .tcl to select the input collection (we will use run 12125 again):
```
   Kanga_tt> cd workdir
   workdir> getdata -t kanga -b12125 -e12125 -v
```
This will produce the file kanga_12125-12125.tcl, which will be source'd as above.)

Now we're ready to run a few events:

   workdir> betarun ../BetaUser/kanga.tcl

When we get the tcl prompt, we must select the run, and change the magnetic field setting (this is a temporary fix). Enter the following:
```
   > source kanga_12125-12125.tcl
   > module talk RooBuildEnv 
   RooBuildEnv>   FixedFieldStrength set 1.51007 
   RooBuildEnv> exit 
```
Also, to avoid a bunch of annoying messages you can type:
```
   > module disable DefaultKlongMicroSelection
```
but there's no harm if you don't.
Now we can continue. Let's try 100 events this time:
```
   > ev begin -nev 100
   > exit
```

Now you can inspect the ROOT output file:

   workdir> bbrroot
   ... 
   root [0] in = TFile("framework.root");      (read in the file)
   root [1] in.ls();                           (list the histograms and ntuples)
   root [2] h1d1->Draw();                      (display a histogram; those are "ones," not lower-case "L"'s.)
   root [3] .q                                 (exit Root)

Exercise 2 is done.

Exercise 3. Producing a more complicated and longer output file

In this exercise we will:

prepare a more complicated analysis based on Exercise 2;
run it in batch;
and produce a ROOT output file.

We will be picking up the sample analysis JpsiK0sMicroFilter.cc from the BetaExamples package. By adding the package, you will be able to examine the routine.
```
   workdir> cd ..
   Kanga_tt> addpkg BetaExamples
```
We need to modify four routines in BetaUser: AppUserBuild.cc, myAnalysis.tcl, kanga.tcl (which we will rename kanga_batch.tcl), and bin_BetaUser.mk. We will do this below.

However, you can skip this if you want: already modified versions of these files may be found in $BFROOT/www/doc/tutorials/15May2000_BetaKanga_Tutorial/kanga_batch. If you'd rather, just copy the contents of this directory into BetaUser and then skip to the compile and link step below. Otherwise switch to the BetaUser directory
```
 >  cd BetaUser
```
and begin editing:
- Edit AppUserBuild.cc. This is where we specify which analysis routines should be linked into the job. Two lines need changing here.
  Replace:
```
   #include "BetaUser/NTrkExample.hh"
```
  with:
```
   //#include "BetaUser/NTrkExample.hh"
   #include "BetaExamples/JpsiK0sMicroFilter.hh"
```
  and:
```
   add( new WorkBook1( "NtrkExample", "trks/evt" ) );
```
  with:
```
   //add( new WorkBook1( "NTrkExample", "trks/evt" ) );
   // Filter using micro info.
   add( new JpsiK0sMicroFilter( "MicroFilter", "filtering with micro info" ) );
```
  The name "MicroFilter" is arbitrary, but it must match the entry in myAnalysis.tcl below.
- Edit bin_BetaUser.mk. To ensure that the linker can find the routine from the BetaExamples package, a line must be added to bin_BetaUser.mk. Just at the end of the file, before:
```
   # BetaSequences provides the Beta-specific makefiles
   #include BetaSequences/BetaExecutable.mk
```
  add:
```
   override LINK_BetaExamples      += $(PACKAGE)GNUmakefile
```
- Edit myAnalysis.tcl. The JpsiK0sMicroFilter routine must be added to the analysis sequence. This is done in myAnalysis.tcl. Replace:
```
   sequence append MyAnalysis WorkBook1
```
  with:
```
   #sequence append MyAnalysis WorkBook1
   sequence append MyAnalysis MicroFilter
```
- Copy kanga.tcl to kanga_batch.tcl and then edit kanga_batch.tcl.
```
 BetaUser>  cp kanga.tcl kanga_batch.tcl
```
  The preceding edits were required to incorporate our new analysis routine. Changes to kanga_batch.tcl are specifically to turn our interactive job into a batch job. We add code to kanga_batch.tcl to reproduce what we typed interactively before. Add, at the end of the file:
```
   module talk RooBuildEnv
      FixedFieldStrength set 1.51007
   exit

   module disable DefaultKlongMicroSelection

   source kanga_12125-12125.tcl

   ev beg -nev 1000

   exit
```
  and save it. Here we assume that the relevant variables have not been changed since Exercise 2.
The edits are done. Switch back to the main test release directory:
```
 BetaUser>  cd ..
```

Now compile and link just as before:

   Kanga_tt> bsub -q bldrecoq -o Exercise3.log gmake lib

when it completes, do

   Kanga_tt> bsub -q bldrecoq -o Exercise3.log gmake BetaUser.bin

or, again, if you're brave, you can do both at once:

   Kanga_tt> bsub -q bldrecoq -o Exercise3.log gmake lib BetaUser.bin

and save yourself one "PENDING".

Now go the the workdir directory.
```
   Kanga_tt> cd workdir
```
To run the job, type:
```
   workdir> bsub -q kanga -o Exercise3.log ./betarun ../BetaUser/kanga_batch.tcl
```
The batch queue kanga is specially set up to have access to a local copy of the kanga datasets, and therefore runs more efficiently.
Note: To run a batch job reading an Objectivity collection, follow a similar procedure, but the job must be submitted to the bfobjy batch queue. Also the collection name must be specified using one of the methods suitable for the BdbEventInput module as in Exercise 1.

The output of this job is, again, the file framework.root. If we don't have time today to actually run the job, a copy of this output file can be found in $BFROOT/www/doc/tutorials/15May2000_BetaKanga_Tutorial/Exercise3.root.

To look at the output:

   workdir> bbrroot
   ... 
   root [0] in = TFile("framework.root");      (read in the file)
   root [1] in.ls();                           (list the histograms and ntuples)
   root [2] h1d3->Draw();                      (display a 1-d histogram)
   root [3] h2d4->Draw();                      (display a 2-d histogram)
               etc...
   root [n] .q                                 (exit Root)

You can use this file in the ROOT tutorial to learn more about ROOT.

Ray F. Cowan

Leon Rochester

Last modified: Thu Jul 6 11:21:20 PDT 2000