<!--#include virtual="header.txt"-->

<h1>Multi-Category Security (MCS) Guide</h1>

<h2 id="overview">Overview<a class="slurm_link" href="#overview"></a></h2>
<p>The MCS Plugin is meant to extend the current Slurm functionality related to
job node <b>exclusivity</b> and job/node information display <b>privacy</b>.

<p>Slurm's <b>OverSubscribe</b> option controls the ability of a partition to
execute more than one job at a time on each resource, no matter "what type"
of job. Slurm job submission clients can also use the <b>--exclusive</b> and
<b>--oversubscribe</b> parameters to request how the job can be
<a href="cons_tres_share.html">shared</a>. The <b>ExclusiveUser</b>
slurm.conf parameter and the --exclusive=<b>user</b> client parameter value
modifies the exclusivity functionality. With this parameter enabled, the
"type of job" now matters when considering exclusivity, so jobs can share
resources <b>based on</b> job users, meaning that only jobs whose user is the
same can share resources. This indeed adds a new dimension to how Slurm manages
exclusivity. With the introduction of the MCS Plugin, Slurm can now be
configured to associate an <b>MCS_label</b> to jobs and optionally ensure that
nodes can only be shared among jobs having the same label.
This adds even more degrees of freedom to how Slurm manages exclusivity,
providing end users with much more flexibility in this area.</p>

<p>Slurm also has the <b>PrivateData</b> <a href="slurm.conf.html">slurm.conf</a>
parameter, which is used to control what type of information is hidden from
regular users. Similar to the exclusivity property, the MCS Plugin also
extends the <b>privacy</b> by filtering job and/or node information
based on the user's access to their <b>MCS_label</b>. This means that privacy
is now less restrictive, and information is not just hidden or not to regular
users, but now it is filtered <b>depending</b> on these configurable/requestable
labels in coordination with the PrivateData option.

<h3 id="limitations">Limitations<a class="slurm_link" href="#limitations"></a>
</h3>

<p>Use of MCS restricts the operation of <a href="preempt.html">Preemption</a>.
Specifically, any job that has an MCS label and requests node exclusivity
according to that label will be prevented from preempting or being preempted by
any job that doesn't match that label. If
<code>MCSParameters=enforced,select</code> is set, these restrictions will
apply to all jobs.</p>

<h2 id="config">Configuration<a class="slurm_link" href="#config"></a></h2>
<p>Two parameters are currently available to configure MCS: <b>MCSPlugin</b>
and <b>MCSParameters</b>.

<ul>
<li><b>MCSPlugin</b> Specifies which plugin should be used. Plugins are
mutually exclusive, and the type of label to be associated depends
on the loaded plugin.</li><br>
<ul>
<li><b>mcs/none</b> is the default and disables MCS labels and functionality.
</li><br>
<li><b>mcs/account</b> MCS labels can only take a value equal to the job's
--account. <b>NOTE</b>: this option requires enabling of accounting.
</li><br>
<li><b>mcs/group</b> MCS labels can only take a value equal to the job's user
group.
</li><br>
<li><b>mcs/user</b> MCS labels can only take a value equal to the username of
the job's --uid.
</li><br>
<li><b>mcs/label</b> MCS labels are arbitrary strings. Use of a job_submit
plugin is recommended to manage job submissions to these labels, as no
enforcement is provided otherwise.
</li><br>
</ul>
</ul>

<p>MCS_labels on jobs can be displayed through 'squeue' with the format option
<b>mcslabel</b> or through 'scontrol show job'. Nodes also acquire a MCS_label,
which is inherited from the allocated job's MCS_label. The node's label can be
displayed with 'scontrol show nodes'. The 'sview' command can also be used to
see these MCS_labels.</p>

<p>MCS_labels on jobs can be updated through 'scontrol update job' with the
field option <b>MCSLabel</b>. Only PENDING jobs can be modified.

<p>Users may either request a particular category label for a job (through the
<b>--mcs-label</b> option), or use the default value generated by the specific
MCS plugin implementation. So labels can be configured to be enforced or set
on demand, and the specific MCS Plugin is responsible for checking the validity
of these labels. When enforced, the MCS Plugin implementation will always
associate a MCS label to a submitted job, either the default value or the one
requested by the user (if it's considered correct).</p>

<p>The selection (exclusivity) of nodes can be filtered on MCS labels either
on demand (ondemand) or always (select) or never (noselect). Users can force
the filter with the <b>--exclusive=mcs</b> option (except if noselect mode is
enabled).
</p>

<p>Label enforcement, node selection filtering policy, private data based on
labels and a list of user groups allowed to be mapped to MCS labels can be
configured through MCSParameters option.</p>

<ul>
<li><b>MCSParameters</b> Specifies the options to pass to the specific MCS
Plugin implementation. Options should satisfy the following expression:
<br>
"[ondemand|enforced][,noselect|select|ondemandselect][,privatedata]:[mcs_plugin_parameters]".
The defaults are "ondemand,ondemandselect" and no privatedata.
</li>
<br>
<ul>
<li><b>ondemand|enforced</b> set MCS label on jobs either on demand (using
--mcs-label option) or always</li>
<li><b>noselect|select|ondemandselect</b> select nodes with filter on MCS
label: never, always or on demand (using --exclusive=mcs option)</li>
<li><b>privatedata</b> accordingly with PrivateData specific option:<br>
<ul>
<li>
If configured with <i>privatedata</i> and <i>PrivateData=jobs</i>, job
information is filtered based on their MCS labels.<br>
</li>
<li>
If configured with <i>privatedata</i> and <i>PrivateData=nodes</i>, node
information is filtered based on their MCS labels.<br>
</li>
<li>
Warning: configuring mcs/label with <i>privatedata</i> disables most filtering
provided by <i>PrivateData=[jobs|nodes]</i> as all users are presumed to have
access to all labels.<br>
</li>
</ul>
<li><b>mcs_plugin_parameters</b> Only mcs/group is currently supporting this
option. It can be used to specify the list of user groups (separated by the
'<b>|</b>' character) that are allowed to be mapped to MCS labels by the
mcs/group plugin.</li>
</li>
</ul>
</li>
</ul>

<p>When enforcing privatedata with mcs/account, users will need to have an
association with just the relevant Account to see other jobs in that Account.
If a Partition is included in the association, it will be over-constrained
and will not allow them to view other jobs in the Account.</p>

Different requests and configurations lead to different combinations of
use-cases. The following table is intended to help understand the end user
the expected behavior (related to exclusivity) for a subset of these use-cases:
<br>
<br>

<table style="page-break-inside: avoid; font-family: Arial,Helvetica,sans-serif;" border="1" bordercolor="#000000" cellpadding="6" cellspacing="0" width="100%">
	<tr>
		<td bgcolor="#e0e0e0">
		        Node filtering:
		</td>
		<td bgcolor="#e0e0e0">
			Label enforcement: <b>ondemand</b><br>
			(MCS_label set only if requested.)
		</td>
		<td bgcolor="#e0e0e0">
			Label enforcement: <b>enforced</b><br>
			(MCS_label is mandatory.)
		</td>
	</tr>
	<tr>
		<td bgcolor="#e0e0e0">
			<b>noselect</b>
		</td>
		<td>
		        No filter on nodes even if --exclusive=mcs requested.
		</td>
		<td>
			No filter on nodes even if --exclusive=mcs requested.
		</td>
	</tr>
	<tr>
		<td bgcolor="#e0e0e0">
			<b>select</b>
		</td>
		<td>
			Filter on nodes <b>only</b> if job MCS_label is set.
		</td>
		<td>
			Always filter on nodes.
		</td>
	</tr>
	<tr>
		<td bgcolor="#e0e0e0">
			<b>ondemandselect</b>
		</td>
		<td>
			Filter on nodes <b>only</b> if --exclusive=mcs.
		</td>
		<td>
			Filter on nodes <b>only</b> if --exclusive=mcs.
		</td>
	</tr>
</table>

<h2 id="examples">Some examples<a class="slurm_link" href="#examples"></a></h2>

<p>slurm.conf:
<pre>
MCSPlugin=mcs/account
MCSParameters=enforced,select,privatedata
</pre>

or

<p>
<pre>
MCSPlugin=mcs/group
MCSParameters=ondemand,noselect:groupA|groupB|groupC
</pre>

or

<p>
<pre>
MCSPlugin=mcs/user
MCSParameters=enforced,select,privatedata
</pre>

or

<p>
<pre>
MCSPlugin=mcs/label
MCSParameters=ondemand,select
</pre>

<h2 id="view_params">View MCS parameters
<a class="slurm_link" href="#view_params"></a>
</h2>

<p>Configured MCS parameters can be viewed using the <i>scontrol</i> command.
<pre>
scontrol show config |grep MCS
MCSPlugin          = mcs/group
MCSParameters      = ondemand,noselect:groupA|groupB|groupC
</pre>

<h3 id="add_label">Put an mcs_label on a job using salloc, sbatch or srun
<a class="slurm_link" href="#add_label"></a>
</h3>
<pre>
srun -n10 -t 1000 --mcs-label=groupB ./job &
</pre>

<h3 id="exclusive">Put an mcs_label on a job with exclusivity
<a class="slurm_link" href="#exclusive"></a>
</h3>
<pre>
srun -n10 -t 1000 --mcs-label=groupB --exclusive=mcs ./job &
</pre>

<h3 id="change_label">
Put a different mcs_label on a job with the mcs/account plugin
<a class="slurm_link" href="#change_label"></a>
</h3>
<pre>
srun -n10 -t 1000 -A another_account_than_default ./job &
</pre>

<h2 id="view_usage">View MCS parameters in usage
<a class="slurm_link" href="#view_usage"></a>
</h2>

<p>The <i>squeue</i> command can also display the MCS with mcslabel
format option
<pre>
squeue -O jobid,username,mcslabel
JOBID               USER                MCSLABEL
2                   foo                 groupA
3	            bar                 groupB
</pre>

<p>The <i>scontrol</i> command will now display the MCS_label
<pre>
scontrol show nodes
NodeName=node0001 Arch=x86_64 CoresPerSocket=4
   CPUAlloc=0 CPUTot=8 CPULoad=0.01 Features=(null)
   Gres=(null)
   NodeAddr=noder0001 NodeHostName=node0001 Version=15.08
   OS=Linux RealMemory=23780 AllocMem=0 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=40 Owner=N/A MCS_label=groupA
   BootTime=2015-08-05T10:14:41 SlurmdStartTime=2015-09-07T13:42:28
   CurrentWatts=0 AveWatts=0
</pre>

<p style="text-align:center;">Last modified 20 January 2025</p>

<!--#include virtual="footer.txt"-->