| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
Jobs can be submitted with optional pre- and post-execution commands. A pre- or post- execution command is an arbitrary command to run before the job starts or after the job finishes. Pre- and post-execution commands are executed in a separate environment from the job.
[ Top ]
About Pre-Execution and Post-Execution Commands
Each batch job can be submitted with optional pre- and post-execution commands. Pre- and post-execution commands can be any excutable command lines to be run before a job is started or after a job finishes.
Some batch jobs require resources that LSF does not directly support. For example, appropriate pre- and/or post-execution commands can be used to handle various situations:
- Reserving devices like tape drives
- Creating and deleting scratch directories for a job
- Customized scheduling
- Checking availability of software licenses
- Assigning jobs to run on specific processors on SMP machines
By default, the pre- and post-execution commands are run under the same user ID, environment, and home and working directories as the batch job. If the command is not in your normal execution path, the full path name of the command must be specified.
For parallel jobs, the command is run on the first selected host.
Pre-execution commands
Pre-execution commands support job starting decisions which cannot be configured directly in LSF. LSF supports both job-level and queue-level pre-execution.
The pre-execution command returns information to LSF using its exit status. When a pre-execution command is specified, the job is held in the queue until the specified pre- execution command returns exit status zero (0).
If the pre-execution command exits with non-zero status, the batch job is not dispatched. The job goes back to the PEND state, and LSF tries to dispatch another job to that host. While the job is pending, other jobs can proceed ahead of the waiting job. The next time LSF tries to dispatch jobs this process is repeated.
If the pre-execution command exits with a value of 99, the job will not go back to the PEND state, it will exit. This gives you flexibility to abort the job if the pre-execution command fails.
LSF assumes that the pre-execution command runs without side effects. For example, if the pre-execution command reserves a software license or other resource, you must not reserve the same resource more than once for the same batch job.
Post-execution commands
If a post-execution command is specified, then the command is run after the job is finished regardless of the exit state of the job.
Post-execution commands are typically used to clean up some state left by the pre- execution and the job execution. Post-execution is only supported at the queue level.
Job-level commands
The bsub -E option specifies an arbitrary command to run before starting the batch
job. When LSF finds a suitable host on which to run a job, the pre-execution command
is executed on that host. If the pre-execution command runs successfully, the batch job
is started.
Job-level post-execution commands are not supported.
Queue-level commands
In some situations (for example, license checking), it is better to specify a queue-level
pre-execution command instead of requiring every job be submitted with the -E option
of bsub.
Queue-level commands run on the execution host before or after a job from the queue is run.
The LSF administrator uses the PRE_EXEC and POST_EXEC parameters in
lsb.queues to set up queue-level pre- and post-execution commands.
Post-execution job states
Some jobs may not be considered complete until some post-job processing is performed. For example, a job may need to exit from a post-execution job script, clean up job files, or transfer job output after the job completes.
The DONE or EXIT job states do not indicate whether post-processing is complete, so
jobs that depend on processing may start prematurely. Use the post_done and
post_err keywords on the bsub -w command to specify job dependency conditions
for job post-processing. The corresponding job states POST_DONE and POST_ERR
indicate the state of the post-processing.
The bhist command displays the POST_DONE and POST_ERR states. The
resource usage of post-processing is not included in the job resource usage.
After the job completes, you cannot perform any job control on the post-processing. Post-processing exit codes are not reported to LSF. The post-processing of a repetitive job cannot be longer than the repetition period.
[ Top ]
Configuring Pre- and Post-Execution Commands
Pre- and post-execution commands can be configured at the job level or on a per-queue basis.
Job-level commands
Job-level pre-execution commands require no configuration. Use the bsub -E option
to specify an arbitrary command to run before the job starts.
The following example shows a batch job that requires a tape drive. The user program
tapeCheck exits with status zero if the specified tape drive is ready:
% bsub -E "/usr/share/bin/tapeCheck /dev/rmt01" myJob
Queue-level commands
Use the PRE_EXEC and POST_EXEC keywords in the queue definition
(lsb.queues) to specify pre- and post-execution commands.
The following points should be considered when setting up pre- and post-execution commands at the queue level:
- If the pre-execution command exits with a non-zero exit code, then it is considered to have failed and the job is requeued to the head of the queue. This feature can be used to implement customized scheduling by having the pre-execution command fail if conditions for dispatching the job are not met.
- Other environment variables set for the job are also set for the pre- and post-execution commands.
- When a job is dispatched from a queue which has a pre-execution command, LSF will remember the post-execution command defined for the queue from which the job is dispatched. If the job is later switched to another queue or the post-execution command of the queue is changed, LSF will still run the original post-execution command for this job.
-
When the post-execution command is run, the environment variable, LSB_JOBEXIT_STAT, is set to the exit status of the job. See the man page for the
wait(2) command for the format of this exit status. -
The post-execution command is also run if a job is requeued because the job's execution environment fails to be set up, or if the job exits with one of the queue's REQUEUE_EXIT_VALUES.
The LSB_JOBPEND environment variable is set if the job is requeued. If the job's execution environment could not be set up, LSB_JOBEXIT_STAT is set to 0.
See Automatic Job Requeue for more information.
- If both queue and job-level pre-execution commands are specified, the job-level pre-execution is run after the queue-level pre-execution command.
The entire contents of the configuration line of the pre- and post-execution commands
are run under /bin/sh -c, so shell features can be used in the command.
For example, the following is valid:
PRE_EXEC = /usr/share/lsf/misc/testq_pre >> /tmp/pre.out POST_EXEC = /usr/share/lsf/misc/testq_post | grep -v "Hey!"
The pre- and post-execution commands are run in /tmp.
Standard input and standard output and error are set to /dev/null. The output from
the pre- and post-execution commands can be explicitly redirected to a file for
debugging purposes.
The PATH environment variable is set to:
PATH='/bin /usr/bin /sbin /usr/sbin'
The pre- and post-execution commands are run under cmd.exe /c.
Standard input and standard output and error are set to NULL. The output from the pre- and post-execution commands can be explicitly redirected to a file for debugging purposes.
See LSB_PRE_POST_EXEC_USER parameter (lsf.sudoers).
See the Platform LSF Reference for information about the lsf.sudoers file.
The following queue specifies the pre-execution command
/usr/share/lsf/pri_prexec and the post-execution command
/usr/share/lsf/pri_postexec.
Begin Queue QUEUE_NAME = priority PRIORITY = 43 NICE = 10 PRE_EXEC = /usr/share/lsf/pri_prexec POST_EXEC = /usr/share/lsf/pri_postexec End Queue
LSB_PRE_POST_EXEC_USER parameter (lsf.sudoers)
By default, both the pre- and post-execution commands are run as the job submission
user. Use the LSB_PRE_POST_EXEC_USER parameter in lsf.sudoers to specify
a different user ID for queue-level pre- and post-execution commands.
For example, if the pre- or post-execution commands perform privileged operations that require root permission, specify:
LSB_PRE_POST_EXEC_USER=root
See the Platform LSF Reference for information about the lsf.sudoers file.
[ Top ]
| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
|
Date Modified: June 06, 2005
Platform Computing: www.platform.com Platform Support: support@platform.com Platform Information Development: doc@platform.com Copyright © 1994-2005 Platform Computing Corporation. All rights reserved. |