Running ATLAS Workbook at SLAC

It is useful to run the ATLAS workbook at CERN first to get familiarized with the general ATLAS software setup, then try the workbook from the SLAC environment. You also need to take note of the SLAC ATLAS computing environment information to recognize the differences between the current SLAC setup and CERN, including disk space for ATLAS work at SLAC. First some general points about using Atlas software at SLAC:

For first time SLAC computing users, you need to first register as a SLAC user and obtain a SLAC computing account. The initial account setup related info can be found on the US Western Tier-2 page under the link "SLAC UNIX users", which also contains links to info on batch system and printing etc. Other general computing services and information provided by the SLAC Scientific Computing and Computing Services (SCCS) can be found at the general "SLAC computing environment" page. There is also some information on configuring ssh to avoid using passwords multiple times. Among the various Public Machines at SLAC, the recommended interactive machines to login are yakut's (yakut@slac.stanford.edu), which are Linux nodes running SL3.
For code development on working packages, the default operating setup is to use the CERN CVS repository over AFS. The Kerberos setup is not yet established at SLAC so that you need to make a simple manual environment definition for CVSROOT (see the `CVSROOT' section below) in your login. This also implies that you must have a CERN account setup already. In case you have not yet done so, some orientations about setting up CERN account may help.
The current SLAC site installation script restricts the user private release setup to a fixed location of ~username/reldirs at present which must be followed in order to get the right library path. You can make different release directories under reldirs with their independent InstallArea. However, there is a file ATLCURRENT in your home directory (~username) which determines your choice of working release.
We recommend ignoring all GRID related setup/examples in the workbook at the first pass and only get back to them later if you wish.
The releases currently available at SLAC are: 12.0.3, 12.0.4, 12.0.5, 12.0.6, 12.3.0, 12.5.0, 13.0.10, and production caches 12.0.6.4 and 12.0.6.5.

Explicit Instructions for Running Athena at SLAC

Kept up by Zach Marshall; you're always welcome to email me if you have trouble or suggestions for improving these insturctions.

First Time Only Set Up

Create a folder called reldirs in your home directory.

mkdir ~/reldirs

Create the file ATLCURRENT in your home directory, a text file that reads "13.0.10" (no quotes) for release 13.0.10, for example.
Add the following line to your .login files (csh for a c-shell or .sh for a borne-based shell):

source /afs/slac.stanford.edu/g/atlas/etc/atlas_env.csh

Follow these instructions to setup remote CERN ssh keys (or you have to type a password eight times for each checkout).
If you wish to set up Hepix look here for instructions.
Add the following line to your login script if your CERN userid is different from your SLAC userid for a c-shell:

setenv CVSROOT :ext:[your username at CERN]@atlas-sw.cern.ch:/atlascvs

Or for a bourne-based shell, the command is:
- export CVSROOT=:ext:[your username at CERN]@atlas-sw.cern.ch:/atlascvs
Make sure there is not a cmthome directory in your home folder. The Atlas scripts will create it on login.

rm -r ~/cmthome

Now follow all the steps under "Using a new release" below.
You should claim yourself some space in the ATLAS scratch space at SLAC

cd ~/reldirs/12.0.3/PhysicsAnalysis/AnalysisCommon/UserAnalysis/run
mkdir /afs/slac.stanford.edu/g/atlas/work/< firstLetterOfUsername >/< username >
ln -s /afs/slac.stanford.edu/g/atlas/work/< firstLetterOfUsername >/< username > Scratch

Using a new release

Create a new release folder within reldirs (if necessary)

mkdir ~/reldirs/12.0.3

Change the file ATLCURRENT in your home directory to the new release number you would like to use
Log out and log back in.
Check out and build the UserAnalysis package to begin with. First you must check the tag number

cmt show versions PhysicsAnalysis/AnalysisCommon/UserAnalysis

Which will return something like:

PhysicsAnalysis/AnalysisCommon/UserAnalysis UserAnalysis-00-08-19 /afs/slac.stanford.edu/g/atlas/b/AtlasAnalysis/2.0.3

This indicates that the tag for the 12.0.3 release (AtlasAnalysis version 2.0.3) is 00-08-19. Now check out the package and build it:

cd ~/reldirs/12.0.3
cmt co -r UserAnalysis-00-08-19 PhysicsAnalysis/AnalysisCommon/UserAnalysis
cd PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
gmake

You should see a library in ~/reldir/12.0.3/InstallArea/i686-slc3-gcc323-opt/lib once this is done

If this is NOT your first release and you wish to copy over old files, you should check out the newest versions of all scripts from the workbook (e.g. jobOptions.G4Atlas_Sim.py) to check for changes in the structure of job options.

To check out additional packages

Check the tag collector to see what the relevant tag is for your package here, or as described above do

cd ~/reldirs/12.0.3
cmt show versions ThisIsWhere/MyPackageSits/MyPackage
cmt co -r MyPackage-xx-yy-zz ThisIsWhere/MyPackageSits/MyPackage

You can now make your package. Once this is done and the libraries appear in your InstallArea, all scripts will use your copy of the package by default.

cd ThisIsWhere/MyPackageSits/MyPackage/cmt
gmake

To look at source code from the packages

Look at the LXR Cross Referencer. You can search it (top right side) for code in the most recent release, and browse source from any of the packages. The options below the "search" options are for different releases that you can browse. Nightly is the most up to date, but the nightlies are not currently available at SLAC (so you may need to check out packages to get this code into your personal build). You can also use the CVS Code Browser.

The Workbook at SLAC

The Atlas workbook should now run without any additional modifications, just as it does at CERN. We offer two suggestions as time savers to those who wish to hurry into analysis. In release 13 one can generate single particles on the fly by editing the default job options file, jobOptions.G4Atlas_Sim.py. In previous releases, the following allows generation of a single particle (considerably faster during simulation than the Z->ee events of the workbook). The second step provides instructions on using the batch system at SLAC.

To Generate your own particles using ParticleGenerator

Change the following lines of your jobOptions.pythia.py file (or a copy of it)

theApp.Dlls += ["ParticleGenerator"] # "TruthExamples", "Pythia_i" ]
theApp.TopAlg = ["ParticleGenerator"] # "Pythia","DumpMC"]
AtRndmGenSvc.Seeds = ["SINGLE 2040160768 443921183" ] # PYTHIA 4789899 989240512", "PYTHIA_INIT 820021 2347532"]

(Comment out the lines with "Pythia = Algorithm("Pythia")" and "Pythia.PythiaCommand")
Add the following lines instead of those

ParticleGenerator = Algorithm( "ParticleGenerator" )
ParticleGenerator.orders = [
"pdgcode: sequence 11 -11",
"energy: fixed 100000",
"vertZ: gauss 0.0 50",
"vertY: vertZ",
"vertX: loop -10 11 1",
"eta: fixed 2.8",
"phi: flat 0 6.2832"
]

In these lines, pdgcode is the pdgcode of the particles you wish to generate (11 is electron, -11 is positron), energy is in MeV, eta is in units of pseudorapidity, and distances are in mm. The various commands include "fixed" for a constant value, "sequence" for alternating between a set of values (this, for instance, will create on electron and then one positron), "gauss" for a gaussian distribution, "flat" for a flat distribution (e.g., this code will shoot randomly in phi), and "loop" for looping through a series of numbers.

To Simulate Events on the Batch System at SLAC

NB: You MUST use the same detector geometry for all of simulation, digitization, and reconstruction. That's the ATLAS-CSC-02-00-00 tag, for example.
To save time, you can create a little bash script for running your simulation on the batch system at SLAC. If you don't run on the batch system, expect to wait 15 minutes per Z to ee event.

cat myBatchJob.bash
#!/bin/bash

athena.py mySimOptions.py G4Atlas_Sim.py > Scratch/athena_sim_Zee.out

You may also have to make your job executable
- chmod ugo+x myBatchJob.bash
Now you can run your job on the batch system. Very few jobs will run in the time allowed by the long queue, so you should almost certainly use the xlong cue.

bsub -q xlong -J MyAtlasSim myBatchJob.bash

Troubleshooting

If you get any of the below problems while working at SLAC, please post them to the Non-Grid Jobs at SLAC forum in ATLAS Hypernews. They should have all been resolved, so these fixes should be unnecessary.

If on log in you get an error like
AtlasLogin: Configuration problem - CMTCONFIG (i686-slc4-gcc345-opt) not available for /afs/slac.stanford.edu/g/atlas/b/AtlasOffline/12.0.3
or something similar upon login (with the atlas_env script), you will need to (you may add this to your .login script):

source ~/cmthome/setup.csh -tag=gcc323,12.0.3 (or .sh for a bourne-based shell)
Or atlsetup 12.5.0,slc3,gcc323 (note: this is a SLAC-specific command)

When running the setup script for the first time, you may get an "Word too long" error. Change your shell to bash to fix the problem.

bash

If generation with Pythia crashes or simulation crashes when Geant retrieves its data (you will be able to tell this from the athena.out file), add the following lines to your setup script for 12.0.3 (or equivalent for another release) at ~/reldirs/12.0.3/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt/setup.sh:

export G4LEVELGAMMADATA=/afs/slac.stanford.edu/g/atlas/releases/vol07/12.0.3/geant4/releases/share/geant4.7.1.patch01.atlas03/data/PhotonEvaporation
export DATAPATH=$ATLROOT/releases/12.0.3/sw/lcg/app/releases/GENSER/GENSER_1_4_0/src/lhapdf/5_0_0/data:$DATAPATH

Or for a c-shell add:

setenv G4LEVELGAMMADATA {$ATLROOT}/releases/vol07/12.0.3/geant4/releases/share/geant4.7.1.patch01.atlas03/data/PhotonEvaporation
setenv DATAPATH {$ATLROOT}/releases/12.0.3/sw/lcg/app/releases/GENSER/GENSER_1_4_0/src/lhapdf/5_0_0/data:$DATAPATH

If package checkout fails with CMT errors, make sure to set:

setenv CVSROOT :ext:atlas-sw.cern.ch:/atlascvs
setenv CVS_RSH ssh

Or for a bourne-based shell

export CVSROOT=:ext:atlas-sw.cern.ch:/atlascvs

Last Update: Jun/19/2007
Su Dong
Zach Marshall