SRS Pipeline: Using the SLAC Batch Farm

If you need more cycles than you readily have your hands on, you're welcome to use the SLAC linux batch farm. The resources of the SLAC linux batch farm give you the ability to run multiple jobs concurrently. The batch system is easy to use and offers substantially more CPU horsepower than can be obtained from the handful of interactive servers available. However, there are some important issues regarding the use of the batch farm as described below.

Test Drive. You can submit a simple batch job, consisting only of the UNIX command "hostname", which prints out the name of the host computer on which the job runs.

1. Login to your SLAC Public account.

2. At the prompt, enter: bsub hostname

This will submit a one-command batch job to the 'short' queue, which gives you about 2 minutes of CPU time (on fell-class machine). A message listing your job number and the queue the job was submitted to will be displayed:

Job <206422> is submitted to default queue <short>.
chuckp@noric06 $

3. Check your email for a message with the name of your host machine; it should be similar to this example: email response.

Normally, a batch job is more complex and you will likely submit a shell or python script to perform the desired computation. The syntax for a more typical batch submission might look like this:

$ bsub -q long -R rhel40 myScript.py

This command submits a job to the 'long' queue (about 2 hours CPU time on a fell-class machine) and further specifies that the job be run on a machine running RedHat Enterprise Linux version 4. A log of the batch job is returned to you as email, and any files created will be directed per your script.

Shared Resources

Once you login to afs, your SLAC Public environment enables you to share a great many resources. For example, when you submit a job to the batch farm, the shared resources you are using include the:

• Interactive machines (e.g., noric, rhel4-32, etc.)
  Note: For a complete list, see Public Machines at SLAC.
  
• Batch farm (non-interactive)
• NFS disks (group space)
• AFS disks (home directories and some group space)
• Xroot disks ????? (storage for bulk data)
• Network facilities

Monitoring Resources

Remember, there are many users. When sharing these resources, it is important to avoid overloading them, thereby degrading the response times and causing jobs to fail for everybody. For example, if you are simultaneously running 100's of jobs that are doing a lot of I/O, the server struggles to handle the requests due to the load.

When running jobs on the batch farm, you can — and should — monitor the activity and state of these resources to ensure that you haven't inadvertently overloaded them. Ganglia monitors many of them, and serves as an early alert to a problem. Once you find the relevant monitoring page(s), select an appropriate time period for the plots, and don't forget to refresh the page with new data periodically.

Tip: The following ganglia links enable you to monitor:

• Shared disk resources NFS disks ??????
• AFS servers and disks ??????
• Xroot disks ??????

Disk access is the single most likely bottleneck in a swarm of batch jobs. To avoid having colleagues beating down your door because your jobs are causing problems for them, take the time to do a bit of homework first and then be prepared to *carefully* monitor your project as it ramps up—and be equally prepared to kill those batch jobs of a problem materializes.

First, assess which disks you will be reading and writing from; examples include:

• Your $HOME directory in AFS (e.g., "dot" files, pfiles).
• Other AFS directories you own where code might be kept.
• NFS directories (e.g., /nfs/farm/g/glast/uXX and /afs/slac/g/glast/users).??????
• Xroot directories (e.g., results of a DataCatalog query). ??????

Next, discover which servers are involved:

1. For AFS disks, use something like this:

$ fs whereis /afs/slac/g/glast/ground/releases/analysisFiles/
File /afs/slac/g/glast/ground/releases/analysisFiles/ is on host afs00.slac.stanford.edu

the server is "afs00".

2. For NFS disks, use something like this:

$ cd /nfs/farm/g/glast/u33
$ pwd
/a/sulky36/g.glast.u33

the server is "sulky36".

3. For xroot, one does not know (even from day-to-day) on which server a particular file is stored on; so, you will just need to monitor the entire system for overloading.

Then, bring up the Ganglia web pages mentioned above, and zero in on the server(s) of interest.

Finally, how do you know if you are stressing the system? Before you begin to submit jobs, take a look at the CPU utilization and the disk and network I/O plots for each server to get an idea of the instantaneous baseline.

1. As your jobs begin to run, any abrupt and significant increase in those two metrics indicates a significant load.

How much is too much? That's hard to say exactly, but here is where the situation can become very painful both to your jobs and to other users:

CPU > 90%
Disk I/O > ~50 MB/s on a single disk

2. If you notice a large number of "nfs_server_badcalls", that is evidence of problems with the server and – if correlated with your batch jobs – it is probably your fault!

If a problem occurs, contact ?????????.

Best Practices

There are some basic guidelines to keep in mind before you submit a batch job.

• Put analysis code and scripts in your afs home directories (which are backed up), and ultimately put your output files in nfs, e.g., the user disk:

1. Create a unique directory in /scratch for your batch job, e.g.,
   
         mkdir -p /scratch/<userid>/${LSB_JOBID}

2. Define this directory as your $HOME and then go there prior to running any ScienceTools/Ftools/etc.,

export HOME=/scratch/<userid>/${LSB_JOBID}
cd ${HOME}

This will automatically take care of PFILES being unique for your job and avoid overloading the /nfs user disk with large numbers of opens and closes.  Create any new files in $HOME and then copy anything you wish to save at the end of your job.

3. Cleanup the scratch directory at end of job (after you have copied out anything you want to save),
        
         rm -rf /scratch/<userid>/${LSB_JOBID}

Note: Cleaning up the scratch directory is critical! Any scratch file left behind will slowly fill up the /scratch partition and eventually fill it up.

• Submitting jobs. While submitting a few jobs at a time will likely result in reasonably rapid turn-around, if you plan to run a large number of batch jobs (e.g., 50 or more), you run the risk of delays and competing with other users of the batch farm. Allocations and job scheduling are complex formulas, making it difficult to predict batch response.

  Tip: Weekends are often good for such work — except just before major conferences when there may be a lot of contention for CPU cycles.

• If you are planning a large batch operation, please inform and coordinate with SAS management (Richard Dubois).

Commands You Need to Know

When using the batch farm, you need to know:

bsub bjobs (-l) bkill bqueues

submit a job get a summary of running jobs (-l gets a longer summary) kill an errant job get info on the queues

Logging into a noric system is your first step. Jobs submitted from these machines will automatically go to the batch farm.

Submitting a job

The syntax is simple:

bsub -q [queueName] <command>

Monitoring a Job

bjobs [-l]

Provides a summary of running jobs.

Tip: It also gives you a batch id for the job, which you can use with bkill.

bpeek <job-id>

Provides a 'peek' at the log file for a running batch job.

Tip: There are 'man' pages for all the batch commands, e.g., 'man bsub'. The batch system is formally known as LSF (Load Sharing Facility), and a brief overview to the system can be read with 'man lsfintro'.

Other useful batch monitoring commands include:

Error Codes on exit

Exit codes from batch jobs should be interpreted similarly to those from any other
unix program. Codes:

• 1-128 are exit codes generated by the job itself.
• 129-255 usually indicate that a signal was received and the value is the signal+128.

For example, an exit code of 131 means the job received signal 3 (= 131-128) which is SIGQUIT.

Note: A brief summary of the signal codes can be seen with the command: kill -l

Queue Information

bqueues (-l)

The standard queues we can submit to are:

Killing a Bad Job

bkill <id>

Assuming a batch id of 217304, the command to kill a bad job would be:

bkill 217304

Note: An id = 0 is a wildcard. To cancel all of your jobs, enter: bkill 0

Queue Limits and CPU Factors