doc/html/power_save.shtml

<!--#include virtual="header.txt"-->

<h1>Slurm Power Saving Guide</h1>

<h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>

<ul>
  <li><a href="#overview">Overview</a></li>
  <li><a href="#config">Configuration</a></li>
  <li><a href="#lifecycle">Node Lifecycle</a></li>
  <li><a href="#manual">Manual Power Saving</a></li>
  <li><a href="#resume_suspend">Resume and Suspend Programs</a></li>
  <li><a href="#tolerance">Fault Tolerance</a></li>
  <li><a href="#images">Booting Different Images</a></li>
  <li><a href="#allocations">Use of Allocations</a></li>
  <li><a href="#nodefeatures">Node Features</a></li>
  <li><a href="#hybrid">Hybrid Cluster</a></li>
  <li><a href="#accounting">Cloud Accounting</a></li>
</ul>

<h2 id="overview">Overview<a class="slurm_link" href="#overview"></a></h2>

<p>Slurm provides an integrated mechanism for nodes being suspended (powered
down, placed into power saving mode) and resumed (powered up, restored from
power saving mode) on demand or by request. When automatic power saving is
enabled, nodes that remain <i>IDLE</i> for <b>SuspendTime</b> are suspended by
<b>SuspendProgram</b>, and nodes needed to complete work allocated to them are
resumed by <b>ResumeProgram</b>. Nodes suspended by <b>SuspendProgram</b> will
be unavailable for scheduling for <b>SuspendTimeout</b> without manual
intervention. Nodes resumed by <b>ResumeProgram</b> must register within
<b>ResumeTimeout</b> or they will become <i>DOWN</i> and their allocated jobs
are requeued. The rate of nodes being resumed or suspended can be controlled by
<b>ResumeRate</b> and <b>SuspendRate</b>.</p>

<p>Slurm can be configured to accomplish power saving by managing compute
resources in any cloud provider (e.g. <a href="http://aws.amazon.com/">Amazon
Web Services</a>, <a href="https://cloud.google.com/">Google Cloud
Platform</a>, <a href="https://azure.microsoft.com/">Microsoft Azure</a>) via
their API. These resources can be combined with an existing cluster to
process excess workload (cloud bursting) or it can operate as an independent
and self-contained cluster.</p>

<p>To enable automatic Power Saving operations in Slurm, you must configure the
following. These programs, timeouts, and rates are not required for manual
power saving operations.</p>

<ul>
  <li><b>ResumeProgram</b> and <b>SuspendProgram</b> must be defined. Each
  must be either a valid path to a program or the name of a
<a href="#PowerAction">PowerAction</a> defined in
  <i>slurm.conf</i>.</li>

  <li><b>ResumeTimeout</b> and <b>SuspendTimeout</b> must be defined, either
  globally or on at least one partition.</li>

  <li><b>SuspendTime</b> must be defined, either globally, on at least one
  partition, or on at least one node, and not be <i>INFINITE</i> or
  <i>-1</i>.</li>

  <li><b>ResumeRate</b> and <b>SuspendRate</b> must be greater than or equal
  to <i>0</i>.</li>
</ul>

<p>Changes in the configuration parameters (e.g. <b>SuspendTime</b>) will take
effect after modifying the <i>slurm.conf</i> configuration file and executing
<code>scontrol reconfigure</code> or restarting the Slurm control daemon,
<i>slurmctld</i>.
</p>

<p>Some runtime changes are the exception and take effect immediately:</p>
<ul>
  <li>A per-node <b>SuspendTime</b> set with <code>scontrol create node</code>,
  <code>scontrol update node</code>, or the equivalent REST API request.</li>
  <li>A dynamic node registered with <code>slurmd -Z</code>.</li>
  <li>A change to a partition's node list.</li>
</ul>

<p>When the rest of the power save configuration is valid, as described below,
a finite <b>SuspendTime</b> from one of these changes enables automatic Power
Saving operations for those nodes without a <code>scontrol reconfigure</code>,
even when the global <b>SuspendTime</b> is <i>INFINITE</i>.
Nodes stop suspending as soon as their <b>SuspendTime</b> is back to
<i>INFINITE</i>, but automatic power save itself stays enabled until the next
<code>scontrol reconfigure</code> or <i>slurmctld</i> restart.
</p>

<p>A per-node <b>SuspendTime</b> needs the rest of the power save
configuration to be valid: <b>SuspendProgram</b>, <b>ResumeProgram</b>,
<b>SuspendRate</b>, <b>ResumeRate</b>,
<b>SlurmctldParameters</b>=<b>power_save_interval</b>, and
<b>SlurmctldParameters</b>=<b>power_save_min_interval</b>.
When it is not, <i>slurmctld</i> handles the request as follows:</p>
<ul>
  <li>A node update with <code>scontrol update node</code> or the REST API is
  rejected for any value, including <i>INFINITE</i>.</li>
  <li>A node creation with <code>scontrol create node</code> or the REST API is
  rejected for a finite value.</li>
  <li>A dynamic node registered with <code>slurmd -Z</code> is accepted.
  <i>slurmctld</i> logs a warning, and the node's <b>SuspendTime</b> has no
  effect until power save is configured and <i>slurmctld</i> is reconfigured or
  restarted.</li>
</ul>

<p>The <a href="scontrol.html">scontrol</a> man page lists the exact
requirements and how long a per-node <b>SuspendTime</b> lasts across a
reconfigure or restart.
</p>

<h2 id="config">Configuration<a class="slurm_link" href="#config"></a></h2>

<p>The following configuration parameters of interest include:</p>

<dl>
  <dt id="DebugFlags"><b>DebugFlags</b><a class="slurm_link" href=
  "#DebugFlags"></a>
  </dt>

  <dd>
    <p>Defines specific subsystems which should provide more detailed event
    logging. Options of interest include:</p>

    <dl>
      <dt id="Power"><b>Power</b><a class="slurm_link" href="#Power"></a>
      </dt>

      <dd>
        <p>Power management plugin and power save (suspend/resume programs)
        details.</p>
      </dd>
    </dl>
  </dd>

  <dt id="PowerAction"><b>PowerAction</b><a class="slurm_link" href=
  "#PowerAction"></a>
  </dt>

  <dd>
    <p>Defines a named power management program, including the location where
    it runs. Once defined, a <b>PowerAction</b>'s name can be used in place of
    a program path for <b>ResumeProgram</b>, <b>ResumeFailProgram</b>,
    or <b>SuspendProgram</b>. A specific <b>PowerAction</b> can also be
    selected for an individual request with the
    <code>Action=&lt;name&gt;</code> option to <code>scontrol power up|down</code>
    (see <a href="#manual">Manual Power Saving</a>).</p>

    <p><b>PowerAction</b> is also supported for reboot operations, but those
    are outside the scope of this guide.</p>

    <pre>
PowerAction=&lt;name&gt; Location=&lt;slurmctld|slurmd&gt; Program="&lt;path&gt; [arguments...]"
</pre>

    <dl>
      <dt><b>PowerAction=&lt;name&gt;</b></dt>
      <dd><p>Unique name for the action.</p></dd>

      <dt><b>Location=slurmctld|slurmd</b></dt>
      <dd><p>Where the program will run. <i>slurmctld</i> runs the program
      on the controller as <b>SlurmUser</b>; <i>slurmd</i> runs the
      program on each affected compute node as <b>SlurmdUser</b>. Power
      up actions must use <i>slurmctld</i>, since powered down nodes cannot
      run a <i>slurmd</i> action.</p></dd>

      <dt><b>Program=&lt;path&gt;</b></dt>
      <dd><p>Program to be executed. Arguments may be included in this field.
      If arguments are included, quote the complete <b>Program</b> value in
      <i>slurm.conf</i>. The program must be accessible and executable by the
      user appropriate for the configured <b>Location</b>. The first
      whitespace-separated token is used as the executable path; any remaining
      tokens are passed as arguments before the arguments added by
      Slurm.</p></dd>
    </dl>

    <p>Slurm appends action-specific arguments after any arguments configured
    in <b>Program</b>:</p>

    <div>
        <table class="tlist">
        <thead>
          <tr>
            <th>Operation</th>
            <th><b>Location=slurmctld</b></th>
            <th><b>Location=slurmd</b></th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td><b>Power Up (ResumeProgram)</b></td>
            <td>Runs with the affected nodes as a hostlist argument.</td>
            <td>Not valid for power up actions.</td>
          </tr>

          <tr>
            <td><b>Power Down (SuspendProgram)</b></td>
            <td>Runs with the affected nodes as a hostlist argument.</td>
            <td>Runs once per affected node. The local node name is passed
            as an argument and is also available in the environment as
            <b>SLURM_NODE_NAME</b>.</td>
          </tr>
        </tbody>
      </table>
    </div>

    <p>Example:</p>

    <pre>
PowerAction=cloud_resume Location=slurmctld Program=/usr/sbin/slurm_resume
PowerAction=node_suspend Location=slurmd    Program="/usr/sbin/shutdown -h now"
ResumeProgram=cloud_resume
SuspendProgram=node_suspend
</pre>
  </dd>

  <dt id="ReconfigFlags"><b>ReconfigFlags</b><a class="slurm_link" href=
  "#ReconfigFlags"></a>
  </dt>

  <dd>
    <p>Flags to control various actions that may be taken when an
    <code>scontrol reconfigure</code> command is issued. Options of interest
    include:</p>

    <dl>
      <dt id="KeepPowerSaveSettings"><b>KeepPowerSaveSettings</b><a class=
      "slurm_link" href="#KeepPowerSaveSettings"></a>
      </dt>

      <dd>
        <p>If set, an <code>scontrol reconfigure</code> command will preserve
        the current state of <b>SuspendExcNodes</b>, <b>SuspendExcParts</b>,
        and <b>SuspendExcStates</b>.</p>
      </dd>
    </dl>
  </dd>

  <dt id="ResumeFailProgram"><b>ResumeFailProgram</b><a class="slurm_link" href=
  "#ResumeFailProgram"></a>
  </dt>

  <dd>
    <p>Program to be executed when nodes fail to resume by
    <b>ResumeTimeout</b>. The argument to the program will be the names of
    the failed nodes (using Slurm's hostlist expression format). This can
    alternatively be set to the name of a
    <a href="#PowerAction">PowerAction</a>.</p>
  </dd>

  <dt id="ResumeProgram"><b>ResumeProgram</b><a class="slurm_link" href=
  "#ResumeProgram"></a>
  </dt>

  <dd>
    <p>Program to be executed to restore nodes from power saving mode. The
    program executes as <i>SlurmUser</i> (as configured in
    <i>slurm.conf</i>). The argument to the program will be the names of
    nodes to be restored from power savings mode (using Slurm's hostlist
    expression format). This can alternatively be set to the name of a
    <a href="#PowerAction">PowerAction</a>.</p>

    <p>If the <i>slurmd</i> daemon fails to respond within the configured
    <b>ResumeTimeout</b> value with an updated BootTime, the node will be
    placed in a DOWN state and the job requesting the node will be requeued.
    If the node isn't actually rebooted (e.g. when multiple-slurmd is
    configured) you can start slurmd with the "-b" option to report the node
    boot time as now.</p>

    <p>A job to node mapping is available in JSON format by reading the
    temporary file specified by the <b>SLURM_RESUME_FILE</b> environment
    variable. This file should be used at the beginning of
    <b>ResumeProgram</b> - see the <a href="#tolerance">Fault Tolerance</a>
    section for more details. This program may use the <code>scontrol show
    nodename</code> command to ensure that a node has booted and the
    <i>slurmd</i> daemon started.</p>

    <pre>
SLURM_RESUME_FILE=/proc/1647372/fd/7:
{
  "all_nodes_resume" : "cloud[1-3,7-8]",
  "jobs" : [
    {
      "exclusive" : "NO",
      "extra" : "An arbitrary string from --extra",
      "features" : "c1,c2",
      "job_id" : 140814,
      "layout" : "tpu7x=2x2x4",
      "nodes_alloc" : "cloud[1-4]",
      "nodes_resume" : "cloud[1-3]",
      "oversubscribe" : "OK",
      "partition" : "cloud",
      "reservation" : "resv_1234"
    },
    {
      "exclusive" : "NO",
      "extra" : null,
      "features" : "c1,c2",
      "job_id" : 140815,
      "layout" : null,
      "nodes_alloc" : "cloud[1-2]",
      "nodes_resume" : "cloud[1-2]",
      "oversubscribe" : "OK",
      "partition" : "cloud",
      "reservation" : null
    },
    {
      "exclusive" : "NODE",
      "extra" : null,
      "features" : null,
      "job_id" : 140816,
      "layout" : null,
      "nodes_alloc" : "cloud[7-8]",
      "nodes_resume" : "cloud[7-8]",
      "oversubscribe" : "NO",
      "partition" : "cloud_exclusive",
      "reservation" : null
    }
  ]
}
</pre>

    <p><b>oversubscribe</b> is NO, YES, or OK; <b>exclusive</b> is NO, NODE,
    USER, MCS, or TOPO (same meanings as <a href="squeue.html#OPT_%h">squeue
    --format</a> <code>%h</code> and
    <a href="squeue.html#OPT_Exclusive">squeue --Format Exclusive</a>. See also
    <code>SLURM_JOB_OVERSUBSCRIBE</code> and <code>SLURM_JOB_EXCLUSIVE</code> in
    <a href="prolog_epilog.html">prolog_epilog</a>.</p>

    <p><b>layout</b> is the layout of the job, as set with
    <a href="sbatch.html#OPT_layout">--layout</a>, or null when the job has
    none.</p>

    <p><b>NOTE</b>: The <b>SLURM_RESUME_FILE</b> will only exist and be
    usable if Slurm was compiled with the <a href=
    "related_software.html#json">JSON-C</a> serializer library.</p>
  </dd>

  <dt id="ResumeRate"><b>ResumeRate</b><a class="slurm_link" href=
  "#ResumeRate"></a>
  </dt>

  <dd>
    <p>Maximum number of nodes to be removed from power saving mode per
    minute. A value of zero results in no limits being imposed. The default
    value is 300. Use this to prevent rapid increases in power
    consumption.</p>
  </dd>

  <dt id="ResumeTimeout"><b>ResumeTimeout</b><a class="slurm_link" href=
  "#ResumeTimeout"></a>
  </dt>

  <dd>
    <p>Maximum time permitted (in seconds) between when a node resume request
    is issued and when the node is actually available for use. Nodes which
    fail to respond in this time frame will be marked DOWN and the jobs
    scheduled on the node requeued. Nodes which reboot after this time frame
    will be marked DOWN with a reason of "Node unexpectedly rebooted." The
    default value is 60 seconds.</p>
  </dd>

  <dt id="SlurmctldParameters"><b>SlurmctldParameters</b><a class="slurm_link"
  href="#SlurmctldParameters"></a>
  </dt>

  <dd>
    <p>Comma-separated options identifying slurmctld options. Options of
    interest include:</p>

    <dl>
      <dt id="cloud_dns"><b>cloud_dns</b><a class="slurm_link" href=
      "#cloud_dns"></a>
      </dt>

      <dd>
        <p>By default, Slurm expects that the network addresses for cloud
        nodes won't be known until creation of the node and that Slurm will
        be notified of the node's address upon registration. Since Slurm
        communications rely on the node configuration found in the
        slurm.conf, Slurm will tell the client command, after waiting for all
        nodes to boot, each node's IP address. However, in environments where
        the nodes are in DNS, this step can be avoided by configuring this
        option.</p>
      </dd>

      <dt id="fast_power_up_launch"><b>fast_power_up_launch</b><a class=
      "slurm_link" href="#fast_power_up_launch"></a>
      </dt>

      <dd>
        <p>Launch a job that is waiting on nodes to power up or reboot as soon
        as the last of its nodes registers, rather than on the next periodic
        poll of the slurmctld background thread (which can add up to 30 seconds
        of delay). This covers jobs waiting on a reboot requested with
        <code>sbatch --reboot</code> or by a node feature change, as well as
        jobs waiting on nodes to resume from power save. Each registration of a
        powering-up node scans the job list, so sites that simultaneously power
        up very large numbers of nodes while holding a very large number of
        jobs may prefer to leave this disabled. Unset by default.</p>
      </dd>

      <dt id="idle_on_node_suspend"><b>idle_on_node_suspend</b><a class=
      "slurm_link" href="#idle_on_node_suspend"></a>
      </dt>

      <dd>
        <p>Mark nodes as idle, regardless of current state, when suspending
        nodes with <b>SuspendProgram</b> so that nodes will be eligible to be
        resumed at a later time.</p>
      </dd>

      <dt id="node_reg_mem_percent"><b>node_reg_mem_percent=#</b><a class=
      "slurm_link" href="#node_reg_mem_percent"></a>
      </dt>

      <dd>
        <p>Percentage of memory a node is allowed to register with without
        being marked as invalid with low memory. Default is 100. For
        State=CLOUD nodes, the default is 90.</p>
      </dd>

      <dt id="power_save_interval"><b>power_save_interval=#</b><a class=
      "slurm_link" href="#power_save_interval"></a>
      </dt>

      <dd>
        <p>How often the power_save thread looks to resume and suspend nodes.
        The power_save thread will do work sooner if there are node state
        changes. Default is 10 seconds.</p>
      </dd>

      <dt id="power_save_min_interval"><b>power_save_min_interval=#</b><a class=
      "slurm_link" href="#power_save_min_interval"></a>
      </dt>

      <dd>
        <p>How often the power_save thread, at a minimum, looks to resume and
        suspend nodes. Default is 0.</p>
      </dd>
    </dl>
  </dd>

  <dt id="SuspendExcNodes"><b>SuspendExcNodes</b><a class="slurm_link" href=
  "#SuspendExcNodes"></a>
  </dt>

  <dd>
    <p>Nodes not subject to suspend/resume logic. This may be used to avoid
    suspending and resuming nodes which are not in the cloud. Alternately the
    suspend/resume programs can treat local nodes differently from nodes
    being provisioned from cloud. Use Slurm's hostlist expression to identify
    nodes with an optional ":" separator and count of nodes to exclude from
    the preceding range. For example <code>nid[10-20]:4</code> will prevent 4
    usable nodes (i.e IDLE and not DOWN, DRAINING or already powered down) in
    the set <code>nid[10-20]</code> from being powered down. Multiple sets of
    nodes can be specified with or without counts in a comma separated list
    (e.g <code>nid[10-20]:4,nid[80-90]:2</code>). By default, no nodes are
    excluded. Nodes explicitly requested to power down, including nodes
    allocated from a partition configured with
    <a href="#PowerDownOnIdle">PowerDownOnIdle=YES</a>, are not affected by
    this option. This value may be updated with scontrol. See
    <b>ReconfigFlags=KeepPowerSaveSettings</b> for setting persistence.</p>
  </dd>

  <dt id="SuspendExcParts"><b>SuspendExcParts</b><a class="slurm_link" href=
  "#SuspendExcParts"></a>
  </dt>

  <dd>
    <p>List of partitions with nodes to never place in power saving mode.
    Multiple partitions may be specified using a comma separator.
    By default, no nodes are excluded. Nodes explicitly requested to power down,
    including nodes allocated from a partition configured with
    <a href="#PowerDownOnIdle">PowerDownOnIdle=YES</a>, are not affected by
    this option. This value may be updated with scontrol. See
    <b>ReconfigFlags=KeepPowerSaveSettings</b> for setting persistence.</p>
  </dd>

  <dt id="SuspendExcStates"><b>SuspendExcStates</b><a class="slurm_link" href=
  "#SuspendExcStates"></a>
  </dt>

  <dd>
    <p>Specifies node states that are not to be powered down automatically.
    Valid states include CLOUD, DEGRADED, DOWN, DRAIN, DYNAMIC_FUTURE,
    DYNAMIC_NORM, FAIL, INVALID_REG, MAINTENANCE, NOT_RESPONDING, PLANNED, and
    RESERVED.
    By default, any of these states, if idle for
    <b>SuspendTime</b>, would be powered down. Nodes explicitly requested to
    power down, including nodes allocated from a partition configured with
    <a href="#PowerDownOnIdle">PowerDownOnIdle=YES</a>, are not affected by
    this option. This value may be updated with scontrol. See
    <b>ReconfigFlags=KeepPowerSaveSettings</b> for setting persistence.</p>
  </dd>

  <dt id="SuspendProgram"><b>SuspendProgram</b><a class="slurm_link" href=
  "#SuspendProgram"></a>
  </dt>

  <dd>
    <p>Program to be executed to place nodes into power saving mode. The
    program executes as <i>SlurmUser</i> (as configured in
    <i>slurm.conf</i>). The argument to the program will be the names of
    nodes to be placed into power savings mode (using Slurm's hostlist
    expression format). This can alternatively be set to the name of a
    <a href="#PowerAction">PowerAction</a>.</p>
  </dd>

  <dt id="SuspendRate"><b>SuspendRate</b><a class="slurm_link" href=
  "#SuspendRate"></a>
  </dt>

  <dd>
    <p>Maximum number of nodes to be placed into power saving mode per
    minute. A value of zero results in no limits being imposed. The default
    value is 60. Use this to prevent rapid drops in power consumption.</p>
  </dd>

  <dt id="SuspendTime"><b>SuspendTime</b><a class="slurm_link" href=
  "#SuspendTime"></a>
  </dt>

  <dd>
    <p>Nodes becomes eligible for power saving mode after being idle or down
    for this number of seconds. A negative number disables power saving mode.
    The default value is -1 (disabled).</p>
  </dd>

  <dt id="SuspendTimeout"><b>SuspendTimeout</b><a class="slurm_link" href=
  "#SuspendTimeout"></a>
  </dt>

  <dd>
    <p>Maximum time permitted (in second) between when a node suspend request
    is issued and when the node shutdown is complete. At that time the node
    must ready for a resume request to be issued as needed for new workload.
    The default value is 30 seconds.</p>
  </dd>
</dl>

<h3 id="node_config">Node Configuration<a class="slurm_link" href=
"#node_config"></a></h3>

<p>Node parameters of interest include:</p>

<dl>
  <dt id="Feature"><b>Feature</b><a class="slurm_link" href="#Feature"></a>
  </dt>

  <dd>
    <p>A node feature can be associated with resources acquired from the
    cloud and user jobs can specify their preference for resource use with
    the <code>--constraint</code> option.</p>
  </dd>

  <dt id="NodeName"><b>NodeName</b><a class="slurm_link" href=
  "#NodeName"></a>
  </dt>

  <dd>
    <p>This is the name by which Slurm refers to the node. A name containing
    a numeric suffix is recommended for convenience.</p>
  </dd>

  <dt id="State"><b>State</b><a class="slurm_link" href="#State"></a>
  </dt>

  <dd>
    <p>Nodes which are to be added on demand should have a state of
    <i>CLOUD</i>.</p>
  </dd>

  <dt id="SuspendTime_node"><b>SuspendTime</b><a class="slurm_link" href=
  "#SuspendTime_node"></a>
  </dt>

  <dd>
    <p>Nodes which remain idle or down for this number of seconds will be
    placed into power saving mode by <b>SuspendProgram</b>.</p>

    <p>This value takes precedence over the partition-level and global
    <b>SuspendTime</b> settings for the specified nodes. If not set on the
    node, the partition and then global values are used. Setting
    <b>SuspendTime</b> to <i>INFINITE</i> will disable suspending of these
    nodes. This value may be updated with
    <code>scontrol update NodeName=&lt;nodename&gt;
    SuspendTime=&lt;seconds|INFINITE|NONE|-1&gt;</code>.</p>
  </dd>

  <dt id="Weight"><b>Weight</b><a class="slurm_link" href="#Weight"></a>
  </dt>

  <dd>
    <p>Each node can be configured with a weight indicating the desirability
    of using that resource. Nodes with lower weights are used before those
    with higher weights. The default value is 1. Slurm will allocate from
    powered up nodes first, powering up nodes second and lastly powered down
    nodes -- respecting node weights in each set.
  </dd>
</dl>

<h3 id="part_config">Partition Configuration<a class="slurm_link" href=
"#part_config"></a></h3>

<p>Partition parameters of interest include:</p>

<dl>
  <dt id="PowerDownOnIdle"><b>PowerDownOnIdle</b><a class="slurm_link" href=
  "#PowerDownOnIdle"></a>
  </dt>

  <dd>
    <p>If set to <i>YES</i> and power saving is enabled for the partition,
    then nodes allocated from this partition will be requested to power down
    after being allocated at least one job. These nodes will not power down
    until they transition from COMPLETING to IDLE. If set to <i>NO</i> then
    power saving will operate as configured for the partition. The default
    value is <i>NO</i>.</p>

    <p>The following will cause a transition from <i>COMPLETING</i> to
    <i>IDLE</i>:</p>

    <ul>
      <li>Completing all running jobs without additional jobs being
      allocated.</li>

      <li><i>Exclusive=USER</i> and after all running jobs complete but
      before another user's job is allocated.</li>

      <li><i>Exclusive=NODE</i> and after the running job completes
      but before another job is allocated.</li>
    </ul>

    <p><b>NOTE</b>: Nodes allocated from this partition will still be sent a
    power down request even if they are excluded from automatic power saving
    mode by <b>SuspendExcNodes</b>, <b>SuspendExcParts</b>, or
    <b>SuspendExcStates</b>.</p>

    <p><b>NOTE</b>: Nodes are still subject to powering down when being IDLE
    for <b>SuspendTime</b> when <b>PowerDownOnIdle</b> is set to <i>NO</i>.</p>
  </dd>

  <dt id="ResumeTimeout_1"><b>ResumeTimeout</b><a class="slurm_link" href=
  "#ResumeTimeout_1"></a>
  </dt>

  <dd>
    <p>Maximum time permitted (in seconds) between when a node resume request
    is issued and when the node is actually available for use. Nodes which
    fail to respond in this time frame will be marked DOWN and the jobs
    scheduled on the node requeued. Nodes which reboot after this time frame
    will be marked DOWN with a reason of "Node unexpectedly rebooted." The
    default value is 60 seconds.</p>

    <p>For nodes that are in multiple partitions with this option set, the
    highest time will take effect. If not set on any partition, the node will
    use the <b>ResumeTimeout</b> value set for the entire cluster.</p>
  </dd>

  <dt id="SuspendTime_1"><b>SuspendTime</b><a class="slurm_link" href=
  "#SuspendTime_1"></a>
  </dt>

  <dd>
    <p>Nodes which remain idle or down for this number of seconds will be
    placed into power saving mode by <b>SuspendProgram</b>.</p>

    <p>For nodes that are in multiple partitions with this option set, the
    highest time will take effect. If not set on any partition, the node will
    use the <b>SuspendTime</b> value set for the entire cluster. A
    <b>SuspendTime</b> value set on a <b>NodeName</b> line takes precedence
    over this partition-level value. Setting <b>SuspendTime</b> to
    <i>INFINITE</i> will disable suspending of nodes in this partition.</p>
  </dd>

  <dt id="SuspendTimeout_1"><b>SuspendTimeout</b><a class="slurm_link" href=
  "#SuspendTimeout_1"></a>
  </dt>

  <dd>
    <p>Maximum time permitted (in second) between when a node suspend request
    is issued and when the node shutdown is complete. At that time the node
    must ready for a resume request to be issued as needed for new workload.
    The default value is 30 seconds.</p>

    <p>For nodes that are in multiple partitions with this option set, the
    highest time will take effect. If not set on any partition, the node will
    use the <b>SuspendTimeout</b> value set for the entire cluster.</p>
  </dd>
</dl>

<h2 id="lifecycle">Node Lifecycle<a class="slurm_link" href=
"#lifecycle"></a></h2>

<p>When Slurm is configured for Power Saving operation, nodes have an
expanded set of states associated with them. States associated with Power
Saving are generally labeled with a symbol when viewing node details with
<code>sinfo</code>.</p>

<div class="figure">
  <img src="node_lifecycle.png" alt="Figure 1. Node Lifecycle" width=
  "1000" /><br />
  Figure 1. Node Lifecycle
</div>

<p>Node states of interest:</p>

<div>
  <table cellpadding="8">
    <tbody>
      <tr>
        <td><u><b>STATE</b></u>
        </td>
        <td style="text-align:center"><b><u>Power Saving Symbol</u></b>
        </td>
        <td><u><b>Description</b></u>
        </td>
      </tr>

      <tr>
        <td>POWER_DOWN</td>
        <td style="text-align:center">!</td>
        <td>Power down request. When the node is no longer running job(s),
        run the <b>SuspendProgram</b>.</td>
      </tr>

      <tr>
        <td>POWER_UP</td>
        <td style="text-align:center">&nbsp;</td>
        <td>Power up request. When possible, run the
        <b>ResumeProgram</b>.</td>
      </tr>

      <tr>
        <td>POWERED_DOWN</td>
        <td style="text-align:center">~</td>
        <td>The node is powered down or in power saving mode.</td>
      </tr>

      <tr>
        <td>POWERING_DOWN</td>
        <td style="text-align:center">%</td>
        <td>The node is in the process of powering down, or being put into
        power saving mode, and is not capable of running any jobs for
        <b>SuspendTimeout</b>.</td>
      </tr>

      <tr>
        <td>POWERING_UP</td>
        <td style="text-align:center">#</td>
        <td>The node is in the process of powering up, or being restored from
        power saving mode.</td>
      </tr>
    </tbody>
  </table>
</div>

<h2 id="manual">Manual Power Saving<a class="slurm_link" href=
"#manual"></a></h2>

<p>A node can be manually powered up and down by setting the state of the node
to the following states using <code>scontrol</code>:</p>

<pre>
scontrol update nodename=&lt;nodename&gt; state=power_&lt;down|down_asap|down_force|up&gt;
</pre>

<p>The equivalent <code>scontrol power</code> subcommand can also be used.
It supports selecting a configured
<a href="#PowerAction">PowerAction</a> for the request instead of the
default <b>ResumeProgram</b> or <b>SuspendProgram</b>:</p>

<pre>
scontrol power &lt;up|down&gt; [asap|force] {ALL|&lt;nodelist&gt;|&lt;nodeset&gt;} [Action=&lt;name&gt;]
[Reason=&lt;reason&gt;]
</pre>

<p><b>NOTE</b>: <code>Reason=</code> is only accepted for
    <code>power down</code>.
</p>

<p><code>power up force</code> will requeue or cancel all jobs on the node,
immediately mark the node as powered down, and then power it up. Unlike
<code>power up</code>, it works on nodes that are already powered up. A DRAIN
flag set by <a href="#POWER_DOWN_ASAP">POWER_DOWN_ASAP</a> is removed. A
DRAIN flag set in any other way is kept.</p>

<p><code>scontrol update</code> command actions/states of interest:</p>

<dl>
  <dt id="POWER_DOWN"><b>POWER_DOWN</b><a class="slurm_link" href=
  "#POWER_DOWN"></a>
  </dt>

  <dd>Will use the configured <b>SuspendProgram</b> program to explicitly
  place a node in power saving mode. If a node is already in the process of
  being powered down, the command will only change the state of the node but
  won't have any effect until the configured <b>SuspendTimeout</b> is
  reached. A power down does not remove a DRAIN flag that the node already
  had, unless <a href="#idle_on_node_suspend">idle_on_node_suspend</a> is
  set.</dd>

  <dt id="POWER_DOWN_ASAP"><b>POWER_DOWN_ASAP</b><a class="slurm_link" href=
  "#POWER_DOWN_ASAP"></a>
  </dt>

  <dd>Will drain the node and mark it for power down. Currently running jobs
  will complete first and no additional jobs will be allocated to the
  node. The DRAIN flag set by this request is removed when the node starts
  powering down, or when the node is powered up with
  <code>power up force</code>. If the node was already drained before the
  request, or an administrator drains it while the power down is pending, the
  node stays drained.</dd>

  <dt id="POWER_DOWN_FORCE"><b>POWER_DOWN_FORCE</b><a class="slurm_link" href=
  "#POWER_DOWN_FORCE"></a>
  </dt>

  <dd>Will cancel all jobs on the node, power it down, and reset its state to
  <i>IDLE</i>.</dd>

  <dt id="POWER_UP"><b>POWER_UP</b><a class="slurm_link" href=
  "#POWER_UP"></a>
  </dt>

  <dd>Will use the configured <b>ResumeProgram</b> program to explicitly move
  a node out of power saving mode. If a node is already in the process of
  being powered up, the command will only change the state of the node but
  won't have any effect until the configured <b>ResumeTimeout</b> is
  reached.</dd>

  <dt id="RESUME"><b>RESUME</b><a class="slurm_link" href="#RESUME"></a>
  </dt>

  <dd>
    <p>Not an actual node state, but will change a node state from DRAIN,
    DRAINING, DOWN or REBOOT to IDLE and NoResp. slurmctld will then attempt
    to contact slurmd to request that the node register itself. Once
    registered, the node state will then remove the NoResp flag and will
    resume normal operations. It will also clear the POWERING_DOWN state of a
    node and make it eligible to be allocated.</p>

    <p><b>NOTE</b>: This does not set <b>NoResp</b> on a node that is
    <b>POWERED_DOWN</b> or <b>POWERING_DOWN</b>. Such a node is neither pinged
    nor able to register while it is powered down, so the flag would persist
    and keep the node out of the available nodes. The node is still required
    to register when it powers up.</p>
  </dd>
</dl>

<h2 id="resume_suspend">Resume and Suspend Programs<a class="slurm_link"
href="#resume_suspend"></a></h2>

<p>When <b>ResumeProgram</b> or <b>SuspendProgram</b> is set to a filesystem
path, that program executes as <i>SlurmUser</i> on the host where the
<i>slurmctld</i> daemon runs (primary and backup controller nodes). When either
parameter names a <a href="#PowerAction">PowerAction</a> instead, the
<b>Location</b> field determines what user runs the program where, as explained
above. The commands used to change power state will depend upon your cluster
management tools and may require <i>sudo</i> for the executing user to run.
If you need to convert Slurm's hostlist expression into individual node names,
the <code>scontrol show hostnames</code> command may prove useful.
</p>

<p>The <b>ResumeProgram</b> and <b>SuspendProgram</b> are not subject to any
time limits but must have <a href="#tolerance">Fault Tolerance</a>. They
should perform the required action, ideally verify the action (e.g. node boot
and start the <i>slurmd</i> daemon, thus the node is no longer non-responsive
to <i>slurmctld</i>) and terminate. Long running programs will be logged by
<i>slurmctld</i>, but not aborted.</p>

<p>Example <b>ResumeProgram</b>:</p>

<pre>
#!/bin/bash
# Example ResumeProgram
hosts=$(scontrol show hostnames "$1")
logfile=/var/log/power_save.log
echo "$(date) Resume invoked $0 $*" &gt;&gt;$logfile
for host in $hosts
do
        sudo node_startup "$host"
done
exit 0
</pre>

<p>Example <b>SuspendProgram</b>:</p>

<pre>
#!/bin/bash
# Example SuspendProgram
hosts=$(scontrol show hostnames "$1")
logfile=/var/log/power_save.log
echo "$(date) Suspend invoked $0 $*" &gt;&gt;$logfile
for host in $hosts
do
        sudo node_shutdown "$host"
done
exit 0
</pre>

<p><b>NOTE</b>: the stderr and stdout of the suspend and resume programs are
not logged. If logging is desired, then it should be added to the
scripts.</p>

<h2 id="tolerance">Fault Tolerance<a class="slurm_link" href=
"#tolerance"></a></h2>

<p>If the <i>slurmctld</i> daemon is terminated gracefully, it will wait up
to ten seconds (or the maximum of <b>SuspendTimeout</b> or
<b>ResumeTimeout</b> if less than ten seconds) for any spawned
<b>SuspendProgram</b> or <b>ResumeProgram</b> to terminate before the daemon
terminates. If the spawned program does not terminate within that time
period, the event will be logged and <i>slurmctld</i> will exit in order to
permit another <i>slurmctld</i> daemon to be initiated. Any spawned
<b>SuspendProgram</b> or <b>ResumeProgram</b> will continue to run.</p>

<p>When the slurmctld daemon shuts down, any <b>SLURM_RESUME_FILE</b>
temporary files are no longer available, even once slurmctld restarts.
Therefore, <b>ResumeProgram</b> should use <b>SLURM_RESUME_FILE</b> within
ten seconds of starting to guarantee that it still exists.</p>

<h2 id="images">Booting Different Images<a class="slurm_link" href=
"#images"></a></h2>

<p>If you want <b>ResumeProgram</b> to boot various images according to job
specifications, it will need to be a fairly sophisticated program and perform
the following actions:</p>

<ol>
  <li>Determine which jobs are associated with the nodes to be booted.
  <b>SLURM_RESUME_FILE</b> will help with this step.</li>

  <li>Determine which image is required for each job. Images can be mapped
  with <a href="#nodefeatures">NodeFeaturesPlugins</a>.
  </li>

  <li>Boot the appropriate image for each node.</li>
</ol>

<h2 id="allocations">Use of Allocations<a class="slurm_link" href=
"#allocations"></a></h2>

<p>A resource allocation request will be granted as soon as resources are
selected for use, possibly before the nodes are all available for use. A job
whose nodes had to power up or reboot does not start until all of them have
registered, and the launching of job steps will be delayed until the required
nodes have been restored to service (it prints a warning about waiting for
nodes to become available and periodically retries until they are
available).</p>

<h2 id="nodefeatures">Node Features<a class="slurm_link" href=
"#nodefeatures"></a></h2>

<p>Features defined by <a href=
"slurm.conf.html#OPT_NodeFeaturesPlugins">NodeFeaturesPlugins</a>, and
associated to cloud nodes in the <b>slurm.conf</b>, will be available but not
active when the node is powered down. If a job requests available but not
active features, the controller will allocate nodes that are powered down and
have the features as available. At allocation, the features will be made
active. A cloud node will remain with the active features until the node is
powered down (i.e. the node can't be rebooted to get other features until the
node is powered down). When the node is powered down, the those features
become available but not active. Any feature not defined by
NodeFeaturesPlugins are always active.</p>

<p>Example:</p>

<pre>
slurm.conf:
NodeFeaturesPlugins=node_features/helpers

NodeName=cloud[1-5] ... State=CLOUD Feature=f1,f2,l1
NodeName=cloud[6-10] ... State=CLOUD Feature=f3,f4,l2

helpers.conf:
NodeName=cloud[1-5] Feature=f1,f2 Helper=/bin/true
NodeName=cloud[6-10] Feature=f3,f4 Helper=/bin/true
</pre>

<p>Features f1, f2, f3, and f4 are changeable features and are defined on the
node lines in the slurm.conf because <i>CLOUD</i> nodes do not register
before being allocated. By setting the <i>Helper</i> script to /bin/true, the
slurmd's will not report any active features to the controller and the
controller will manage all the active features. If the <i>Helper</i> is set
to a script that reports the active features, the controller will validate
that the reported active features are a super set of the node's active
changeable features in the controller. Features l1 and l2 will always be
active and can be used as selectable labels.</p>

<h2 id="hybrid">Hybrid Cluster<a class="slurm_link" href="#hybrid"></a></h2>

<p>Cloud nodes to be acquired on demand can be placed into their own Slurm
partition. This mode of operation can be used to use these nodes only if so
requested by the user. Note that jobs can be submitted to multiple partitions
and will use resources from whichever partition permits faster initiation. A
sample configuration in which nodes are added from the cloud when the
workload exceeds available resources. Users can explicitly request local
resources or resources from the cloud by using the <code>--constraint</code>
option.</p>

<p>Example:</p>

<pre>
# Excerpt of slurm.conf
SelectType=select/cons_tres
SelectTypeParameters=CR_CORE_Memory

SuspendProgram=/usr/sbin/slurm_suspend
ResumeProgram=/usr/sbin/slurm_resume
SuspendTime=600
SuspendExcNodes=tux[0-127]
TreeWidth=128

NodeName=DEFAULT    Sockets=1 CoresPerSocket=4 ThreadsPerCore=2
NodeName=tux[0-127] Weight=1 Feature=local State=UNKNOWN
NodeName=ec[0-127]  Weight=8 Feature=cloud State=CLOUD
PartitionName=debug MaxTime=1:00:00 Nodes=tux[0-32] Default=YES
PartitionName=batch MaxTime=8:00:00 Nodes=tux[0-127],ec[0-127]
</pre>

<p>When <b>SuspendTime</b> is set globally, Slurm attempts to suspend all
nodes unless excluded by <b>SuspendExcNodes</b> or <b>SuspendExcParts</b>. It
can be tricky to have to remember to add on-premise nodes to the excluded
options. By setting the global <b>SuspendTime</b> to <i>INFINITE</i> and
configuring <b>SuspendTime</b> on cloud specific partitions, you can avoid
having to exclude nodes.</p>

<p>Example:</p>

<pre>
# Excerpt of slurm.conf
SelectType=select/cons_tres
SelectTypeParameters=CR_CORE_Memory

SuspendProgram=/usr/sbin/slurm_suspend
ResumeProgram=/usr/sbin/slurm_resume
TreeWidth=128

NodeName=DEFAULT    Sockets=1 CoresPerSocket=4 ThreadsPerCore=2
NodeName=tux[0-127] Weight=1 Feature=local State=UNKNOWN
NodeName=ec[0-127]  Weight=8 Feature=cloud State=CLOUD
PartitionName=debug MaxTime=1:00:00 Nodes=tux[0-32] Default=YES
PartitionName=batch MaxTime=8:00:00 Nodes=tux[0-127],ec[0-127]
PartitionName=cloud Nodes=ec[0-127] SuspendTime=600
</pre>

<p>Here we have configured a partition with only cloud nodes and defined
<b>SuspendTime</b> on that partition. Doing so will allow us to control when
those nodes power down without affecting our on-premise nodes, therefore
<b>SuspendExcNodes</b> or <b>SuspendExcParts</b> are not needed in this
setup.</p>

<h2 id="accounting">Cloud Accounting<a class="slurm_link" href=
"#accounting"></a></h2>

<p>Information about cloud instances can be stored in the database. This can
be done by configuring instance id/type upon slurmd startup or with
<i>scontrol update</i>. The node's "extra" field will also be stored in the
database.</p>

<p>Configuring cloud information on slurmd startup:</p>

<pre>
$ slurmd --instance-id=12345 --instance-type=m7g.medium --extra="arbitrary string" . . .
</pre>

<p>Configuring cloud information with scontrol update:</p>

<pre>
$ scontrol update nodename=n1 instanceid=12345 instancetype=m7g.medium extra="arbitrary string"
</pre>

<p>This data can then be seen on the controller with <i>scontrol show
node</i>. Past and current data can be seen in the database with <i>sacctmgr
show instance</i>, as well as through slurmrestd with the <i>/instance</i>
and <i>/instances</i> endpoints.</p>

<p>Showing cloud information on the controller with scontrol:</p>

<pre>
$ scontrol show nodes n1 | grep "NodeName\|Extra\|Instance"
NodeName=n1 Arch=x86_64 CoresPerSocket=4
   Extra=arbitrary string
   InstanceId=12345 InstanceType=m7g.medium
</pre>

<p>Showing cloud information from the database with sacctmgr:</p>

<pre>
$ sacctmgr show instance format=nodename,instanceid,instancetype,extra
NodeName                  InstanceId         InstanceType                Extra
--------------- -------------------- -------------------- --------------------
n1                             12345           m7g.medium     arbitrary string
</pre>

<p>Showing cloud information from the database with slurmrestd:</p>

<pre>
$ curl -k -s \
        --request GET \
        -H X-SLURM-USER-NAME:$(whoami) \
        -H X-SLURM-USER-TOKEN:$SLURM_JWT \
        -H "Content-Type: application/json" \
        --url localhost:8080/slurmdb/v0.0.40/instances \
        | jq ".instances"

[
  {
    "cluster": "c1",
    "extra": "arbitrary string",
    "instance_id": "12345",
    "instance_type": "m7g.medium",
    "node_name": "n1",
    "time": {
      "time_end": 0,
      "time_start": 1687213177
    }
  }
]
</pre>

<h2 id="topology">Topology
<a class="slurm_link" href="#topology"></a>
</h2>

<p>Cloud nodes can have topologies defined statically in the config files, or
can be dynamically added to and removed from topologies as described in the
<a href="topology.html">Topology Guide</a>.</p>

<!--#include virtual="footer.txt"-->