SCons for Fermi/LAT: An Overview

SCons was chosen for its cross-platform capabilities and replaces CMT as the Science Analysis Systems build tool. SCons is Open Source software written in Python, and our Fermi-specific extensions are also written in Python. (For a more detailed introduction, go to http://scons.org/.)

Details of the build system used to produce releases will, for the most part, not be of interest to end-users, but the file organization within an SCons-built release and tools associated with SCons releases, such as the Installer, are. This Overview will concentrate on those aspects of the Fermi LAT software environment as it describes how to acquire a release and run programs belonging to it. Experienced CMT users may wish to consult the CMT – SCons Comparison chart.

Accessing a Release

From a Central SLAC Linux machine

SCons builds of ScienceTools (and most other containers) may be found on the u35 disk:

/nfs/farm/g/glast/u35/ReleaseManagerBuild

SCons builds of GlastRelease are on the u52disk.

Builds are further organized by platform-compiler (e.g. redhat4-i686-32bit-gcc34, redhat5-x86_64-64bit-gcc41, etc.), Debug or Optimized, container name, and finally the top-level directory of a single build. For tagged releases the top-level directory is of the form xx-yy-zz (e.g., 09-18-06). The path to a typical build looks like this:

/nfs/farm/g/glast/u35/ReleaseManagerBuild/redhat4-i686-32bit-gcc34/Optimized/ScienceTools/09-18-06

The general form for the full absolute path for container container-name and name build-name is:

/nfs/farm/g/glast/u35/ReleaseManagerBuild/platform-compiler/dbg-opt/container-name/build-name

Such a release may be used in-place (details to be covered later) or may be copied to your private disk space. Alternatively, you may use one of the Installers to fetch and install a release in your private disk space. See more on Installers in the next section.

From your desktop or laptop

Use one of the Installers for SCons builds, depending on your taste in interfaces: GUI or command-line.

There is a slight difference in the way the GUI and the Command Line installers operate. The GUI installer will create a subdirectory of the directory you specify named after the release (e.g., ScienceTools-3153 for LATEST build 3153) and install there. RMViewer (a tool primarily of interest to developers which also includes an install function) and the command line installer install directly in the directory specified.

With any of these methods you will have a choice of three types of distributions: source, developer or end-user. If you expect to use the release as is, choose end-user. developer will also do, but it includes many more files which you won't need.

Anatomy of a Release Manager Build

An end-user build obtained by using one of the installers has subdirectories lib/build-variant and exe/build-variant for binaries; bin/build-variant for wrapper and set-up scripts. It has several other subdirectories, such as data, xml and python, for platform-independent files. Some of the latter themselves have per-package subdirectories; others don't.

Note: Here build-variant is similar in form to platform-compiler above but unfortunately not identical; it has an additional component to distinguish between debug and optimized builds. Typical examples of build-variant names are redhat4-i686-32bit-Optimized and Windows-i386-32bit-vc90-Debug.

Developer builds have everything included in end-user builds plus per-package subdirectories for sources just as they appear in CVS.

A build in the SLAC SCons build area on u35 has everything a developer build has and in addition, intermediate build products under the per-package subdirectories.

External Libraries: Directory Structure

SCons builds expect the following organization of external libraries:

• A top-level directory, commonly known as GLAST_EXT. Typically the environment variable GLAST_EXT is defined to point to this directory.

• A subdirectory of GLAST_EXT for each supported external, named after the external (e.g. CLHEP, cfitsio, python, ...

• Within each external subdirectory, subdirectories for each available version.

• For some external/version combinations: within the version subdirectory, a subdirectory named after the compiler.

• Within the version (or compiler if it exists) subdirectory, whatever collection of directories is required for that particular external. Typically they have names like include, lib, bin, doc.

External Libraries at SLAC

To access external libraries from a SLAC Linux machine, define GLAST_EXT as:

/afs/slac/g/glast/ground/GLAST_EXT/platform-compiler

e.g.,

/afs/slac/g/glast/ground/GLAST_EXT/redhat4-i686-32bit-gcc34

As of May, 2012, for ScienceTools platform-compiler may be any of:

redhat4-i686-32bit-gcc34
redhat5-i686-32bit-gcc41
redhat5-x86_64-64bit-gcc41
redhat6-x86_64-64bit-gcc44

GlastRelease builds currently exist for redhat4 32-bit Linux and redhat5 32-bit Linux

Windows users with access to the SLAC V disk may set GLAST_EXT to:

V:\Glast_Software\Toaster\GLAST_EXT\Windows-i386-32bit-v71
V:\Glast_Software\Toaster\GLAST_EXT\Windows-i386-32bit-v90

for VS 2003 or VS 2008, respectively. As of May, 2012, the Windows Release Manager builds ScienceTools and GlastRelease (both compilers).

Running Programs from an SCons Build

There are two ways to run executables from an SCons build: by invoking wrapper scripts (in which case a new process is spawned for each program run) or by setting up the environment for a process by means of a special script (_setup.sh or _setup.csh for unix; _setup.bat for Windows) generated as part of the RM build, then invoking executables directly from that process.

Using Wrapper Scripts

Wrapper scripts are in the bin/build-variant subdirectory of the build. A typical session using this approach involves the following steps:

• Set GLAST_EXT environment variable
• (optional) Do any other set-up required for your session; e.g., establish path to ftools
• Run one or more wrapper scripts.
  
  Note: wrapper scripts change (add onto) standard path variables so should be run in a child process. From the unix (Linux or Mac) command line just invoke the file directly. On Windows use the command cmd with the /c option.

unix	myST/bin/redhat4-i686-32bit-gcc34/test_astro
Windows	cmd /c myST\bin\XP-i386-32bit-vc71\test_astro.bat

Using the _setup script

The idea here is to establish the proper environment in a process, then run one or more programs from that process by invoking the executables directly. Executables are located in the exe/build-variant subdirectory of the build. The basic steps for this method are:

• set environment variable GLAST_EXT as above.
• set environment variable INST_DIR to the top-level directory of the build.
• run the setup file.

In this mode the setup file is typically run in the same process in which you plan to run the executable(s). On unix (Linux or Mac) use the source command. Depending on your shell the command will be one of

source $INST_DIR/bin/build-variant/_setup.sh
source $INST_DIR/bin/build-variant/_setup.csh

On Windows simply run the setup file:

%INST_DIR%\bin\build-variant\_setup.bat

Here build-variant is used in the same sense as above; i.e., has a value like redhat4-i686-32bit-gcc34-Optimized on Linux or Windows-i386-32bit-vc71-Debug on Windows.

Optionally do other environment setup needed for your analysis before or after running the SCons-generated setup file.

• Run one or more executables:

unix	myST/exe/redhat4-i686-32bit-gcc34/test_astro
Windows	myST\exe\XP-i386-32bit-vc71\test_astro.exe

Environment variables

The wrapper scripts and the setup script set only a few variables. Most of them are required so that the system will be able to find the executables and libraries belonging to the build at run time. Individual programs may need other resources at run time, such as xml files. The directories holding such files are also typically identifed by means of environment variables, but the files — and hence the variables — are not needed until after the program starts up. They are defined by the program itself, which calls a special initialization routine. If the package myPkg installs resources used by a program (in that package or another one), that program will have defined a suitable environment variable.

There are currently 3 types of resources (xml files, data files, job options files). The corresponding environment variable names for directories where resources belonging to myPkg can be found are:

MYPKGXMLPATH
MYPKGDATAPATH
MYPKGJOBOPTIONSPATH