Affinity preemption

To enable affinity preemption, set PREEMPT_JOBTYPE = AFFINITY in the Parameters section of lsb.params. By default, affinity resources are not preemptable.

Affinity preemption supports the following:
  • Preemption of affinity resources (cores, threads, sockets, NUMA nodes, and NUMA memory)

  • Backfill of reserved affinity resources

  • Pending License Scheduler jobs can use the affinity resources of a suspended License Scheduler job, as long as both jobs request at least one license in common

Affinity preemption interacts with the following LSF features:

Queue-based affinity resource preemption

A running job with affinity requirements may occupy cores in a low priority queue. When affinity preemption is enabled, a pending job in a high priority queue that also has an affinity requirement is potentially able to preempt the running job in the low priority queue to get its affinity resources (threads, cores, sockets, NUMA nodes). When PREEMPTABLE_RESOURCES = mem is enabled in lsb.params a higher priority affinity job can preempt a running low priority job for host memory, NUMA memory as well as slots.

Affinity resources are treated similar to slots and memory: when a job is suspended, the job continues to occupy its slots and its affinity resources, preventing another job from using these resources, unless that other job is in a queue that has a preemption relationship with the suspended job.

Affinity resource backfill

A job in a reservation queue may reserve slots, memory and affinity resources (and potentially other reservable resources). If the reserving job has an affinity requirement, LSF can reserve affinity resources for the job. A job in a backfill queue that has an affinity requirement can use the reserved affinity resources of a pending job if the backfill job is expected to finish before the earliest expected start time of the reserving job. The rule of thumb is that if a job in a backfill queue is able to use the slots reserved by another job during backfill scheduling, then it should be also able to use the reserved affinity resources. Affinity backfill is enabled by default, and cannot be disabled.

License Scheduler affinity resource preemption

In LSF 9.1.1 and License Scheduler 9.1, slots, and optionally, memory are released by a suspended License Scheduler job only to other License Scheduler jobs that request at least one license in common with the suspended job.

This feature also applies to affinity resources. Once a License Scheduler job is suspended, the affinity resources occupied by the job are available to other License Scheduler jobs that request at least one license in common with the suspended job, in its rusage. When LSF_LIC_SCHED_PREEMPT_SLOT_RELEASE=N in lsf.conf, affinity resources along with slots and memory are not released to pending License Scheduler jobs. LSF_LIC_SCHED_PREEMPT_SLOT_RELEASE is enabled by default.

Preemption queue preference

You can configure which queues should have preference to preempt from. Preemption queue preference is enabled by USE_SUSP_SLOTS=Y in lsb.params. USE_SUSP_SLOTS=Y supports affinity preemption. With this parameter enabled, pending jobs in preemptable queues are allowed to use the slots of suspended jobs in higher priority preemptive queues. The queues must have a preemption releationship with each other. When USE_SUSP_SLOTS=N, pending jobs in a low priority preemptable queue cannot use the slots of a suspended job in a high priority preemptive queue.

When USE_SUSP_SLOTS=Y, then pending jobs in preemptable queues are allowed to use the affinity resources occupied by suspended jobs in higher priority preemptive queues, if the queues have a preemption relationship. Note that SSUSP jobs on a host are always allowed to try to resume and use the non-releasable resources, including slots, memory, and affinity resources, occupied by other suspended jobs on the same host.

Memory preemption

By default, LSF considers memory to be a non-releasable resource. When a running job is suspended, LSF continues to reserve memory for the suspended job. When memory preemption is enabled by setting PREEMPTABLE_RESOURCES = mem in lsb.params, jobs with memory requirements submitted to high priority preemptive queues can preempt jobs in low priority queues for memory. When LSF allows jobs in preemptive queues to use memory reserved for suspended jobs, LSF essentially allows host memory to be overcommitted. Host-based memory is a separate resource from the memory reservations made on the NUMA nodes. However, preemption can be triggered for NUMA-level memory as well when memory is configured as a preemptable resource.