<!--#include virtual="header.txt"-->

<h1><a name="top">Generic Resource (GRES) Design Guide</a></h1>

<h2 id="overview">Overview<a class="slurm_link" href="#overview"></a></h2>

<p>Generic Resources (GRES) are resources associated with a specific node
that can be allocated to jobs and steps. The most obvious example of
GRES use would be GPUs. GRES are identified by a specific name and use an
optional plugin to provide device-specific support. This document is meant
to provide details about Slurm's implementation of GRES support including the
relevant data structures. For an overview of GRES configuration and use, see
<a href="gres.html">Generic Resource (GRES) Scheduling</a>.

<h2 id="data">Data Structures<a class="slurm_link" href="#data"></a></h2>

<p>GRES are associated with Slurm nodes, jobs and job steps. You will find
a string variable named <b>gres</b> in those data structures which
is used to store the GRES configured on a node or required by a job or step
(e.g. "gpu:2,nic:1"). This string is also visible to various Slurm commands
viewing information about those data structures (e.g. "scontrol show job").
There is a second variable associated with each of those data structures on
the <b>slurmctld</b> daemon
named <b>gres_list</b> that is intended for program use only. Each element
in the list <b>gres_list</b> provides information about a specific GRES type
(e.g. one data structure for "gpu" and a second structure with information
about "nic"). The structures on <b>gres_list</b> contain an ID number
(which is faster to compare than a string) plus a pointer to another structure.
This second structure differs somewhat for nodes, jobs, and steps (see
<b>gres_node_state_t</b>, <b>gres_job_state_t</b>, and <b>gres_step_state_t</b> in
<b>src/common/gres.h</b> for details), but contains various counters and bitmaps.
Since these data structures differ for various entity types, the functions
used to work with them are also different. If no GRES are associated with a
node, job or step, then both <b>gres</b> and <b>gres_list</b> will be NULL.</p>

<pre>
------------------------
|   Job Information    |
|----------------------|
| gres = "gpu:2,nic:1" |
|      gres_list       |
------------------------
           |
           +---------------------------------
           |                                |
   ------------------               ------------------
   | List Struct    |               | List Struct    |
   |----------------|               |----------------|
   | id = 123 (gpu) |               | id = 124 (nic) |
   |   gres_data    |               |   gres_data    |
   ------------------               ------------------
           |                                |
           |                              ....
           |
           |
------------------------------------------------
| gres_job_state_t                             |
|----------------------------------------------|
| gres_count = 2                               |
| node_count = 3                               |
| gres_bitmap(by node) = 0,1;                  |
|                        2,3;                  |
|                        0,2                   |
| gres_count_allocated_to_steps(by node) = 1;  |
|                                          1;  |
|                                          1   |
| gres_bitmap_allocated_to_steps(by node) = 0; |
|                                           2; |
|                                           0  |
------------------------------------------------
</pre>

<h2 id="op">Mode of Operation<a class="slurm_link" href="#op"></a></h2>

<p>After the slurmd daemon reads the configuration files, it calls the function
<b>node_config_load()</b> for each configured plugin. This can be used to
validate the configuration, for example validate that the appropriate devices
actually exist. If no GRES plugin exists for that resource type, the information
in the configuration file is assumed correct. Each node's GRES information is
reported by slurmd to the slurmctld daemon at node registration time.</p>

<p>The slurmctld daemon maintains GRES information in the data structures
described above for each node, including the number of configured and allocated
resources. If those resources are identified with a specific device file
rather than just a count, bitmaps are used record which specific resources have
been allocated to jobs.</p>

<p>The slurmctld daemon's GRES information about jobs includes several arrays
equal in length to the number of allocated nodes. The index into each of the
arrays is the sequence number of the node in that's job's allocation (e.g.
the first element is node zero of the <b>job</b> allocation). The job step's
GRES information is similar to that of a job including the design where the
index into arrays is based upon the job's allocation. This means when a job
step is allocated or terminates, the required bitmap operations are very
easy to perform without computing different index values for job and step
data structures.</p>

<p>The most complex operation on the GRES data structures happens when a job
changes size (has nodes added or removed). In that case, the array indexed by
node index must be rebuilt, with records shifting as appropriate. Note that
the current software is not compatible with having different GRES counts by
node (a job can not have 2 GPUs on one node and 1 GPU on a second node),
although that might be addressed at a later time.</p>

<p>When a job or step is initiated, its credential includes allocated GRES information.
This can be used by the slurmd daemon to associate those resources with that
job. Our plan is to use the Linux cgroups logic to bind a job and/or its
tasks with specific GRES devices, however that logic does not currently exist.
What does exist today is a pair of plugin APIs, <b>job_set_env()</b> and
<b>step_set_env()</b> which can be used to set environment variables for the
program directing it to GRES which have been allocated for its use (the CUDA
libraries base their GPU selection upon environment variables, so this logic
should work for CUDA today if users do not attempt to manipulate the
environment variables reserved for CUDA use).</p>

<p>If you want to see how GRES logic is allocating resources, configure
<b>DebugFlags=GRES</b> to log GRES state changes. Note the resulting output can
be quite verbose, especially for larger clusters.</p>


<p style="text-align:center;">Last modified 6 August 2021</p>

<!--#include virtual="footer.txt"-->