doc/html/qos.shtml

<!--#include virtual="header.txt"-->

<h1>Quality of Service (QOS)</h1>

<p>One can specify a Quality of Service (QOS) for each job submitted
to Slurm. The QOSs are defined in the Slurm database using the <i>sacctmgr</i>
command. Jobs request a QOS using the "--qos=" option to the
<i>sbatch</i>, <i>salloc</i>, and <i>srun</i> commands.</p>

<h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>
<ul>
<li><a href="#effects">Effects on Jobs</a>
<ul>
<li> <a href=#priority>Scheduling Priority</a>
<li> <a href=#preemption>Preemption</a>
<li> <a href=#limits>Resource Limits</a>
</ul>
<li> <a href=#partition>Partition QOS</a>
<li> <a href=#relative>Relative QOS</a>
<li> <a href=#qos_other>Other QOS Options</a>
<li> <a href=#config>Configuration</a>
<li> <a href=#examples>Examples</a>
</li>
</ul>

<!-------------------------------------------------------------------------->

<h2 id="effects">Effects on Jobs
<a class="slurm_link" href="#priority"></a>
</h2>

<p>The QOS associated with a job will affect the job in three key ways:
scheduling priority, preemption, and resource limits.</p>

<h3 id="priority">Job Scheduling Priority
<a class="slurm_link" href="#priority"></a>
</h3>

<p>Job scheduling priority is made up of a number of factors as
described in the <a
href="priority_multifactor.html">priority/multifactor</a> plugin.  One
of the factors is the QOS priority.  Each QOS is defined in the Slurm
database and includes an associated priority.  Jobs that request and
are permitted a QOS will incorporate the priority associated with that
QOS in the job's <a
href="priority_multifactor.html#general">multi-factor priority
calculation.</a></p>

<p>To enable the QOS priority component of the multi-factor priority
calculation, the "PriorityWeightQOS" configuration parameter must be
defined in the slurm.conf file and assigned an integer value greater
than zero.</p>

<P> A job's QOS only affects is scheduling priority when the
multi-factor plugin is loaded.</P>

<!-------------------------------------------------------------------------->
<h3 id="preemption">Job Preemption
<a class="slurm_link" href="#preemption"></a>
</h3>

<p>Slurm offers two ways for a queued job to preempt a running job,
free-up the running job's resources and allocate them to the queued
job.  See the <a href="preempt.html"> Preemption description</a> for
details.</p>

<p>The preemption method is determined by the "PreemptType"
configuration parameter defined in slurm.conf.  When the "PreemptType"
is set to "preempt/qos", a queued job's QOS will be used to determine
whether it can preempt a running job. It is important to note that the QOS
used to determine if a job is eligible for preemption is the QOS associated
with the job and not a <a href="#partition">Partition QOS</a>.</p>

<P> The QOS can be assigned (using <i>sacctmgr</i>) a list of other
QOSs that it can preempt.  When there is a queued job with a QOS that
is allowed to preempt a running job of another QOS, the Slurm
scheduler will preempt the running job.</P>

<P> The QOS option PreemptExemptTime specifies the minimum run time before the
job is considered for preemption. The QOS option takes precedence over the
global option of the same name. A Partition QOS with PreemptExemptTime
takes precedence over a job QOS with PreemptExemptTime, unless the job QOS
has the OverPartQOS flag enabled.</p>

<!-------------------------------------------------------------------------->
<h3 id="limits">Resource Limits<a class="slurm_link" href="#limits"></a></h3>

<p>Each QOS is assigned a set of limits which will be applied to the
job.  The limits mirror the limits imposed by the
user/account/cluster/partition association defined in the Slurm
database and described in the <a href="resource_limits.html"> Resource
Limits page</a>.  When limits for a QOS have been defined, they
will take precedence over the association's limits.</p>

<!-------------------------------------------------------------------------->
<h2 id="partition">Partition QOS
<a class="slurm_link" href="#partition"></a>
</h2>
<p>A QOS can be attached to a partition. This means the partition will have all
the same limits as the QOS. This does not associate jobs with the QOS, nor does
it give the job any priority or preemption characteristics of the assigned QOS.
Jobs may separately request the same QOS or a different QOS to gain those
characteristics. However, the Partition QOS limits will override the job's QOS.
If the opposite is desired you may configure the job's QOS with
<code>Flags=OverPartQOS</code> which will reverse the order of precedence.</p>

<p>This functionality may be used to implement a true "floating"
partition, in which a partition may access a limited amount of resources with no
restrictions on which nodes it uses to get the resources. This is accomplished
by assigning all nodes to the partition, then configuring a Partition QOS with
<code>GrpTRES</code> set to the desired resource limits.</p>

<p>Most QOS attributes are set using the <b>sacctmgr</b> command. However,
setting a QOS as a partition QOS is accomplished in <b>slurm.conf</b> through
the <a href="slurm.conf.html#OPT_QOS">QOS=</a> option in the configuration of
the associated partition. The QOS should be created using <b>sacctmgr</b> before
it is assigned as a partition QOS. It is possible to delete a QOS attached to a
partition without deleting the attachment in the <b>slurm.conf</b>. This is not
recommended and could cause unintended behavior.</p>

<p>A partition's QOS may also be set, changed or removed at runtime with
<code>scontrol update PartitionName=&lt;name&gt; QOS=&lt;qos&gt;</code>. Running
and suspended jobs apply their usage to the partition's current QOS, so the
update is refused with the error "Partition is in use" while any such job is in
the partition. To change the QOS while such jobs exist, set the new value in
<b>slurm.conf</b> and run <code>scontrol reconfigure</code>, which rebuilds the
QOS usage from the current jobs. Pending jobs do not block a runtime update, and
their submit usage is moved from the old partition QOS to the new one.</p>

<!-------------------------------------------------------------------------->
<h2 id="relative">Relative QOS
<a class="slurm_link" href="#relative"></a>
</h2>
<p>Starting in Slurm 23.11, a QOS may be configured to contain relative resource
limits instead of absolute limits by setting <code>Flags=Relative</code>.
When this flag is set, all resource limits are treated as percentages of the
total resources available. Values higher than 100 are interpreted as 100%.
Memory limits should be set with no units. Although the default units (M) will
be displayed, the limits will be enforced as a percentage (1M = 1%).</p>

<p><b>NOTE</b>: When <i>Flags=Relative</i> is added to a QOS, <b>slurmctld</b>
must be restarted or reconfigured for the flag to take effect.</p>

<p>Generally, the limits on a relative QOS will be calculated relative to the
resources in the whole cluster. For example, <code>cpu=50</code> would be
interpreted as 50% of all CPUs in the cluster.</p>

<p>However, when a relative QOS is also assigned as a partition QOS, some unique
conditions will apply:</p>
<ol>
<li>Limits will be calculated relative to the partition's resources;
for example, <code>cpu=50</code> would be interpreted as 50% of all CPUs in the
associated partition.</li>
<li>Only one partition may have this QOS as its partition QOS.</li>
<li>Jobs will not be allowed to use it as a normal QOS.<br>
<b>NOTE</b>: To avoid unexpected job submission errors, it is recommended not
to add a relative partition QOS to any association-based entities.
</li>
</ol>

<!-------------------------------------------------------------------------->
<h2 id="qos_other">Other QOS Options
<a class="slurm_link" href="#qos_other"></a>
</h2>
<ul>
<li><b>Flags</b> Used by the slurmctld to override or enforce certain
characteristics. To clear a previously set value use the modify command with a
new value of -1.
<br>Valid options are:

<ul>
<li><b>DenyOnLimit</b> If set, jobs using this QOS will be rejected at
submission time if they do not conform to the QOS 'Max' limits as
stand-alone jobs.
Jobs that go over these limits when other jobs are considered, but conform
to the limits when considered individually will not be rejected. Instead they
will pend until resources are available (as by default without DenyOnLimit).
Group limits (e.g. <b>GrpTRES</b>) will also be treated like 'Max' limits
(e.g. <b>MaxTRESPerNode</b>) and jobs will be denied if they would violate the
limit as stand-alone jobs.
This currently only applies to QOS and Association limits.</li>

<li><b>EnforceUsageThreshold</b> If set, and the QOS also has a UsageThreshold,
any jobs submitted with this QOS that fall below the UsageThreshold
will be held until their Fairshare Usage goes above the Threshold.</li>

<li><b>NoDecay</b> If set, this QOS will not have its GrpTRESMins,
GrpWall and RawUsage decayed by the slurm.conf PriorityDecayHalfLife
setting.  This allows
a QOS to provide aggregate limits that, once consumed, will not be
replenished automatically.  Such a QOS will act as a time-limited quota
of resources for an association that has access to it.  Account/user
usage will still be decayed for associations using the QOS.  The QOS
GrpTRESMins and GrpWall limits can be increased or
the QOS RawUsage value reset to 0 (zero) to again allow jobs submitted
with this QOS to run (if pending with QOSGrp{TRES}MinutesLimit or
QOSGrpWallLimit reasons, where {TRES} is some type of trackable resource).
Note that this flag does not affect PriorityUsageResetPeriod, which still
zeroes these counters on its own schedule.
This flag covers the whole QOS; to stop a single TRES decaying while the rest
of the QOS carries on, see
<a href="resource_limits.html#tresdecayhalflife">TresDecayHalfLife</a>, which
this flag takes precedence over.</li>

<li><b>NoReserve</b> If this flag is set and backfill scheduling is used,
jobs using this QOS will not reserve resources in the backfill
schedule's  map of resources allocated through time. This flag is
intended for use with a QOS that may be preempted by jobs associated
with all other QOS (e.g use with a "standby" QOS). If this flag is
used with a QOS which can not be preempted by all other QOS, it could
result in starvation of larger jobs.</li>

<li><b>OverPartQOS</b> If set, jobs using this QOS will be able to
override any limits used by the requested partition's QOS limits.</li>

<li><b>PartitionMaxNodes</b> If set, jobs using this QOS will be able to
override the requested partition's MaxNodes limit.</li>

<li><b>PartitionMinNodes</b> If set, jobs using this QOS will be able to
override the requested partition's MinNodes limit.</li>

<li><b>PartitionTimeLimit</b> If set, jobs using this QOS will be able to
override the requested partition's TimeLimit.</li>

<li><b>Relative</b> If set, the QOS limits will be treated as percentages of
the cluster or partition instead of absolute limits (numbers should be less than
100). The controller should be restarted or reconfigured after adding the
<i>Relative</i> flag to the QOS.
<br>If this is used as a partition QOS:
<ol>
<li>Limits will be calculated relative to the partition's resources.</li>
<li>Only one partition may have this QOS as its partition QOS.</li>
<li>Jobs will not be allowed to use it as a normal QOS.</li>
</ol></li>

<li><b>RequiresReservation</b> If set, jobs using this QOS must designate a
reservation when submitting a job.  This option can be useful in
restricting usage of a QOS that may have greater preemptive capability
or additional resources to be allowed only within a reservation.</li>

<li><b>UsageFactorSafe</b> If set, and <b>AccountingStorageEnforce</b> includes
<b>Safe</b>, jobs will only be able to run if the job can run to completion
with the <b>UsageFactor</b> applied.</li>
</ul>
</li>

<li><b>GraceTime</b> Preemption grace time to be extended to a job
  which has been selected for preemption.</li>

<li><p><b>UsageFactor</b>
A float that is factored into a job's TRES usage (e.g. RawUsage, TRESMins,
TRESRunMins). For example, if the usagefactor was 2, for every TRESBillingUnit
second a job ran it would count for 2. If the usagefactor was .5, every second
would only count for half of the time. A setting of 0 would add no timed usage
from the job.</li>
</p>

<p>
The usage factor only applies to the job's QOS and not the partition QOS.
</p>

<p>
If the <b>UsageFactorSafe</b> flag <b>is</b> set and
<b>AccountingStorageEnforce</b> includes <b>Safe</b>, jobs will only be
able to run if the job can run to completion with the <b>UsageFactor</b>
applied.
</p>

<p>
If the <b>UsageFactorSafe</b> flag is <b>not</b> set and
<b>AccountingStorageEnforce</b> includes <b>Safe</b>, a job will be able to be
scheduled without the <b>UsageFactor</b> applied and will be able to run
without being killed due to limits.
</p>

<p>
If the <b>UsageFactorSafe</b> flag is <b>not</b> set and
<b>AccountingStorageEnforce</b> does not include <b>Safe</b>, a job will be
able to be scheduled without the <b>UsageFactor</b> applied and could be killed
due to limits.
</p>

<p>
See <b>AccountingStorageEnforce</b> in slurm.conf man page.
</p>

<p>
<b>NOTE</b>: A job's TRESRunMins usage is booked using the QOS UsageFactor in
effect when the job starts, and is given back with that same factor when the
job ends. Modifying a QOS's UsageFactor does not rescale the TRESRunMins usage
of jobs that are already running; RawUsage and GrpWall continue to accrue with
the current factor. However, when slurmctld is restarted or reconfigured, the
usage of all running jobs is booked again with the UsageFactor in effect at
that time.
</p>

<p>
Default is 1. To clear a previously set value use the modify command with a new
value of -1.
</p>

<li><b>UsageThreshold</b>
A float representing the lowest fairshare of an association allowable
to run a job.  If an association falls below this threshold and has
pending jobs or submits new jobs those jobs will be held until the
usage goes back above the threshold.  Use <i>sshare</i> to see current
shares on the system.</li>
</ul>

<h2 id="config">Configuration<a class="slurm_link" href="#config"></a></h2>

<P> To summarize the above, the QOSs and their associated limits are
defined in the Slurm database using the <i>sacctmgr</i> utility.  The
QOS will only influence job scheduling priority when the multi-factor
priority plugin is loaded and a non-zero "PriorityWeightQOS" has been
defined in the slurm.conf file.  The QOS will only determine job
preemption when the "PreemptType" is defined as "preempt/qos" in the
slurm.conf file.  Limits defined for a QOS (and described above) will
override the limits of the user/account/cluster/partition
association.</P>

<h2 id="examples">QOS examples<a class="slurm_link" href="#examples"></a></h2>

<p>QOS manipulation examples. All QOS operations are done using
the sacctmgr command. The default output of 'sacctmgr show qos' is
very long given the large number of limits and options available
so it is best to use the format option which filters the display.</p>

<p>By default when a cluster is added to the database a default
qos named normal is created.</p>

<pre>
$ sacctmgr show qos format=name,priority
      Name   Priority
---------- ----------
    normal          0
</pre>

<p>Add a new QOS</p>

<pre>
$ sacctmgr add qos zebra
 Adding QOS(s)
  zebra
 Settings
  Description    = QOS Name

$ sacctmgr show qos format=name,priority
      Name   Priority
---------- ----------
    normal          0
     zebra          0
</pre>

<p>Set QOS priority</p>

<pre>
$ sacctmgr modify qos zebra set priority=10
 Modified qos...
  zebra

$ sacctmgr show qos format=name,priority
      Name   Priority
---------- ----------
    normal          0
     zebra         10
</pre>

<p>Set some other limits</p>

<pre>
$ sacctmgr modify qos zebra set GrpTRES=cpu=24
 Modified qos...
  zebra

$ sacctmgr show qos format=name,priority,GrpTRES
      Name   Priority       GrpTRES
---------- ---------- -------------
    normal          0
     zebra         10        cpu=24
</pre>

<p>Add a QOS to a user account</p>

<pre>
$ sacctmgr modify user crock set qos=zebra

$ sacctmgr show assoc format=cluster,user,qos
   Cluster       User                  QOS
---------- ---------- --------------------
canis_major                          normal
canis_major      root                normal
canis_major                          normal
canis_major     crock                zebra
</pre>

<p>Users can belong to multiple QOSs</p>

<pre>
$ sacctmgr modify user crock set qos+=alligator
$ sacctmgr show assoc format=cluster,user,qos
   Cluster       User                  QOS
---------- ---------- --------------------
canis_major                          normal
canis_major      root                normal
canis_major                          normal
canis_major     crock       alligator,zebra

</pre>

<p>Finally, delete a QOS</p>

<pre>
$ sacctmgr delete qos alligator
 Deleting QOS(s)...
  alligator
</pre>

<!--#include virtual="footer.txt"-->