Chunk Jobs and Job Arrays

Job chunking

LSF supports job chunking, where jobs with similar resource requirements submitted by the same user are grouped together for dispatch. The CHUNK_JOB_SIZE parameter in lsb.queues and lsb.applications specifies the maximum number of jobs allowed to be dispatched together in a chunk job.

Job chunking can have the following advantages:
  • Reduces communication between sbatchd and mbatchd, and scheduling overhead in mbatchd

  • Increases job throughput in mbatchd and more balanced CPU utilization on the execution hosts

All of the jobs in the chunk are dispatched as a unit rather than individually. Job execution is sequential, but each chunk job member is not necessarily executed in the order it was submitted.


You cannot auto-migrate a suspended chunk job member.

Job arrays

LSF provides a structure called a job array that allows a sequence of jobs that share the same executable and resource requirements, but have different input files, to be submitted, controlled, and monitored as a single unit. Using the standard LSF commands, you can also control and monitor individual jobs and groups of jobs submitted from a job array.

After the job array is submitted, LSF independently schedules and dispatches the individual jobs.

Job packs

If your jobs are not related and do not have similar resource requirements, but you still want to submit a large group of jobs quickly and reduce system overhead, you can use the job packs feature instead of job arrays or job chunking.