doc/html/mcs.shtml - SchedMD/slurm - Git at Google

 <!--#include virtual="header.txt"-->

 <h1>Multi-Category Security (MCS) Guide</h1>

 <h2 id="overview">Overview<a class="slurm_link" href="#overview"></a></h2>
 <p>The MCS Plugin is meant to extend the current Slurm functionality related to
 job node <b>exclusivity</b> and job/node information display <b>privacy</b>.

 <p>Slurm's <b>OverSubscribe</b> option controls the ability of a partition to
 execute more than one job at a time on each resource, no matter "what type"
 of job. Slurm job submission clients can also use the <b>--exclusive</b> and
 <b>--oversubscribe</b> parameters to request how the job can be
 <a href="cons_tres_share.html">shared</a>. The <b>ExclusiveUser</b>
 slurm.conf parameter and the --exclusive=<b>user</b> client parameter value
 modifies the exclusivity functionality. With this parameter enabled, the
 "type of job" now matters when considering exclusivity, so jobs can share
 resources <b>based on</b> job users, meaning that only jobs whose user is the
 same can share resources. This indeed adds a new dimension to how Slurm manages
 exclusivity. With the introduction of the MCS Plugin, Slurm can now be
 configured to associate an <b>MCS_label</b> to jobs and optionally ensure that
 nodes can only be shared among jobs having the same label.
 This adds even more degrees of freedom to how Slurm manages exclusivity,
 providing end users with much more flexibility in this area.</p>

 <p>Slurm also has the <b>PrivateData</b> <a href="slurm.conf.html">slurm.conf</a>
 parameter, which is used to control what type of information is hidden from
 regular users. Similar to the exclusivity property, the MCS Plugin also
 extends the <b>privacy</b> by filtering job and/or node information
 based on the user's access to their <b>MCS_label</b>. This means that privacy
 is now less restrictive, and information is not just hidden or not to regular
 users, but now it is filtered <b>depending</b> on these configurable/requestable
 labels in coordination with the PrivateData option.

 <h3 id="limitations">Limitations<a class="slurm_link" href="#limitations"></a>
 </h3>

 <p>Use of MCS restricts the operation of <a href="preempt.html">Preemption</a>.
 Specifically, any job that has an MCS label and requests node exclusivity
 according to that label will be prevented from preempting or being preempted by
 any job that doesn't match that label. If
 <code>MCSParameters=enforced,select</code> is set, these restrictions will
 apply to all jobs.</p>

 <h2 id="config">Configuration<a class="slurm_link" href="#config"></a></h2>
 <p>Two parameters are currently available to configure MCS: <b>MCSPlugin</b>
 and <b>MCSParameters</b>.

 <ul>
 <li><b>MCSPlugin</b> Specifies which plugin should be used. Plugins are
 mutually exclusive, and the type of label to be associated depends
 on the loaded plugin.</li><br>
 <ul>
 <li><b>mcs/none</b> is the default and disables MCS labels and functionality.
 </li><br>
 <li><b>mcs/account</b> MCS labels can only take a value equal to the job's
 --account. <b>NOTE</b>: this option requires enabling of accounting.
 </li><br>
 <li><b>mcs/group</b> MCS labels can only take a value equal to the job's user
 group.
 </li><br>
 <li><b>mcs/user</b> MCS labels can only take a value equal to the username of
 the job's --uid.
 </li><br>
 <li><b>mcs/label</b> MCS labels are arbitrary strings. Use of a job_submit
 plugin is recommended to manage job submissions to these labels, as no
 enforcement is provided otherwise.
 </li><br>
 </ul>
 </ul>

 <p>MCS_labels on jobs can be displayed through 'squeue' with the format option
 <b>mcslabel</b> or through 'scontrol show job'. Nodes also acquire a MCS_label,
 which is inherited from the allocated job's MCS_label. The node's label can be
 displayed with 'scontrol show nodes'. The 'sview' command can also be used to
 see these MCS_labels.</p>

 <p>MCS_labels on jobs can be updated through 'scontrol update job' with the
 field option <b>MCSLabel</b>. Only PENDING jobs can be modified.

 <p>Users may either request a particular category label for a job (through the
 <b>--mcs-label</b> option), or use the default value generated by the specific
 MCS plugin implementation. So labels can be configured to be enforced or set
 on demand, and the specific MCS Plugin is responsible for checking the validity
 of these labels. When enforced, the MCS Plugin implementation will always
 associate a MCS label to a submitted job, either the default value or the one
 requested by the user (if it's considered correct).</p>

 <p>The selection (exclusivity) of nodes can be filtered on MCS labels either
 on demand (ondemand) or always (select) or never (noselect). Users can force
 the filter with the <b>--exclusive=mcs</b> option (except if noselect mode is
 enabled).
 </p>

 <p>Label enforcement, node selection filtering policy, private data based on
 labels and a list of user groups allowed to be mapped to MCS labels can be
 configured through MCSParameters option.</p>

 <ul>
 <li><b>MCSParameters</b> Specifies the options to pass to the specific MCS
 Plugin implementation. Options should satisfy the following expression:
 <br>
 "[ondemand|enforced][,noselect|select|ondemandselect][,privatedata]:[mcs_plugin_parameters]".
 The defaults are "ondemand,ondemandselect" and no privatedata.
 </li>
 <br>
 <ul>
 <li><b>ondemand|enforced</b> set MCS label on jobs either on demand (using
 --mcs-label option) or always</li>
 <li><b>noselect|select|ondemandselect</b> select nodes with filter on MCS
 label: never, always or on demand (using --exclusive=mcs option)</li>
 <li><b>privatedata</b> accordingly with PrivateData specific option:<br>
 <ul>
 <li>
 If configured with <i>privatedata</i> and <i>PrivateData=jobs</i>, job
 information is filtered based on their MCS labels.<br>
 </li>
 <li>
 If configured with <i>privatedata</i> and <i>PrivateData=nodes</i>, node
 information is filtered based on their MCS labels.<br>
 </li>
 <li>
 Warning: configuring mcs/label with <i>privatedata</i> disables most filtering
 provided by <i>PrivateData=[jobs|nodes]</i> as all users are presumed to have
 access to all labels.<br>
 </li>
 </ul>
 <li><b>mcs_plugin_parameters</b> Only mcs/group is currently supporting this
 option. It can be used to specify the list of user groups (separated by the
 '<b>|</b>' character) that are allowed to be mapped to MCS labels by the
 mcs/group plugin.</li>
 </li>
 </ul>
 </li>
 </ul>

 <p>When enforcing privatedata with mcs/account, users will need to have an
 association with just the relevant Account to see other jobs in that Account.
 If a Partition is included in the association, it will be over-constrained
 and will not allow them to view other jobs in the Account.</p>

 Different requests and configurations lead to different combinations of
 use-cases. The following table is intended to help understand the end user
 the expected behavior (related to exclusivity) for a subset of these use-cases:
 <br>
 <br>

 <table style="page-break-inside: avoid; font-family: Arial,Helvetica,sans-serif;" border="1" bordercolor="#000000" cellpadding="6" cellspacing="0" width="100%">
 	<tr>
 		<td bgcolor="#e0e0e0">
 		        Node filtering:
 		</td>
 		<td bgcolor="#e0e0e0">
 			Label enforcement: <b>ondemand</b><br>
 			(MCS_label set only if requested.)
 		</td>
 		<td bgcolor="#e0e0e0">
 			Label enforcement: <b>enforced</b><br>
 			(MCS_label is mandatory.)
 		</td>
 	</tr>
 	<tr>
 		<td bgcolor="#e0e0e0">
 			<b>noselect</b>
 		</td>
 		<td>
 		        No filter on nodes even if --exclusive=mcs requested.
 		</td>
 		<td>
 			No filter on nodes even if --exclusive=mcs requested.
 		</td>
 	</tr>
 	<tr>
 		<td bgcolor="#e0e0e0">
 			<b>select</b>
 		</td>
 		<td>
 			Filter on nodes <b>only</b> if job MCS_label is set.
 		</td>
 		<td>
 			Always filter on nodes.
 		</td>
 	</tr>
 	<tr>
 		<td bgcolor="#e0e0e0">
 			<b>ondemandselect</b>
 		</td>
 		<td>
 			Filter on nodes <b>only</b> if --exclusive=mcs.
 		</td>
 		<td>
 			Filter on nodes <b>only</b> if --exclusive=mcs.
 		</td>
 	</tr>
 </table>

 <h2 id="examples">Some examples<a class="slurm_link" href="#examples"></a></h2>

 <p>slurm.conf:
 <pre>
 MCSPlugin=mcs/account
 MCSParameters=enforced,select,privatedata
 </pre>

 or

 <p>
 <pre>
 MCSPlugin=mcs/group
 MCSParameters=ondemand,noselect:groupA|groupB|groupC
 </pre>

 or

 <p>
 <pre>
 MCSPlugin=mcs/user
 MCSParameters=enforced,select,privatedata
 </pre>

 or

 <p>
 <pre>
 MCSPlugin=mcs/label
 MCSParameters=ondemand,select
 </pre>

 <h2 id="view_params">View MCS parameters
 <a class="slurm_link" href="#view_params"></a>
 </h2>

 <p>Configured MCS parameters can be viewed using the <i>scontrol</i> command.
 <pre>
 scontrol show config |grep MCS
 MCSPlugin          = mcs/group
 MCSParameters      = ondemand,noselect:groupA|groupB|groupC
 </pre>

 <h3 id="add_label">Put an mcs_label on a job using salloc, sbatch or srun
 <a class="slurm_link" href="#add_label"></a>
 </h3>
 <pre>
 srun -n10 -t 1000 --mcs-label=groupB ./job &
 </pre>

 <h3 id="exclusive">Put an mcs_label on a job with exclusivity
 <a class="slurm_link" href="#exclusive"></a>
 </h3>
 <pre>
 srun -n10 -t 1000 --mcs-label=groupB --exclusive=mcs ./job &
 </pre>

 <h3 id="change_label">
 Put a different mcs_label on a job with the mcs/account plugin
 <a class="slurm_link" href="#change_label"></a>
 </h3>
 <pre>
 srun -n10 -t 1000 -A another_account_than_default ./job &
 </pre>

 <h2 id="view_usage">View MCS parameters in usage
 <a class="slurm_link" href="#view_usage"></a>
 </h2>

 <p>The <i>squeue</i> command can also display the MCS with mcslabel
 format option
 <pre>
 squeue -O jobid,username,mcslabel
 JOBID               USER                MCSLABEL
 2                   foo                 groupA
 3	            bar                 groupB
 </pre>

 <p>The <i>scontrol</i> command will now display the MCS_label
 <pre>
 scontrol show nodes
 NodeName=node0001 Arch=x86_64 CoresPerSocket=4
    CPUAlloc=0 CPUTot=8 CPULoad=0.01 Features=(null)
    Gres=(null)
    NodeAddr=noder0001 NodeHostName=node0001 Version=15.08
    OS=Linux RealMemory=23780 AllocMem=0 Sockets=2 Boards=1
    State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=40 Owner=N/A MCS_label=groupA
    BootTime=2015-08-05T10:14:41 SlurmdStartTime=2015-09-07T13:42:28
    CurrentWatts=0 AveWatts=0
 </pre>

 <p style="text-align:center;">Last modified 20 January 2025</p>

 <!--#include virtual="footer.txt"-->
	<!--#include virtual="header.txt"-->

	<h1>Multi-Category Security (MCS) Guide</h1>

	<h2 id="overview">Overview<a class="slurm_link" href="#overview"></a></h2>
	<p>The MCS Plugin is meant to extend the current Slurm functionality related to
	job node <b>exclusivity</b> and job/node information display <b>privacy</b>.

	<p>Slurm's <b>OverSubscribe</b> option controls the ability of a partition to
	execute more than one job at a time on each resource, no matter "what type"
	of job. Slurm job submission clients can also use the <b>--exclusive</b> and
	<b>--oversubscribe</b> parameters to request how the job can be
	<a href="cons_tres_share.html">shared</a>. The <b>ExclusiveUser</b>
	slurm.conf parameter and the --exclusive=<b>user</b> client parameter value
	modifies the exclusivity functionality. With this parameter enabled, the
	"type of job" now matters when considering exclusivity, so jobs can share
	resources <b>based on</b> job users, meaning that only jobs whose user is the
	same can share resources. This indeed adds a new dimension to how Slurm manages
	exclusivity. With the introduction of the MCS Plugin, Slurm can now be
	configured to associate an <b>MCS_label</b> to jobs and optionally ensure that
	nodes can only be shared among jobs having the same label.
	This adds even more degrees of freedom to how Slurm manages exclusivity,
	providing end users with much more flexibility in this area.</p>

	<p>Slurm also has the <b>PrivateData</b> <a href="slurm.conf.html">slurm.conf</a>
	parameter, which is used to control what type of information is hidden from
	regular users. Similar to the exclusivity property, the MCS Plugin also
	extends the <b>privacy</b> by filtering job and/or node information
	based on the user's access to their <b>MCS_label</b>. This means that privacy
	is now less restrictive, and information is not just hidden or not to regular
	users, but now it is filtered <b>depending</b> on these configurable/requestable
	labels in coordination with the PrivateData option.

	<h3 id="limitations">Limitations<a class="slurm_link" href="#limitations"></a>
	</h3>

	<p>Use of MCS restricts the operation of <a href="preempt.html">Preemption</a>.
	Specifically, any job that has an MCS label and requests node exclusivity
	according to that label will be prevented from preempting or being preempted by
	any job that doesn't match that label. If
	<code>MCSParameters=enforced,select</code> is set, these restrictions will
	apply to all jobs.</p>

	<h2 id="config">Configuration<a class="slurm_link" href="#config"></a></h2>
	<p>Two parameters are currently available to configure MCS: <b>MCSPlugin</b>
	and <b>MCSParameters</b>.

	<ul>
	<li><b>MCSPlugin</b> Specifies which plugin should be used. Plugins are
	mutually exclusive, and the type of label to be associated depends
	on the loaded plugin.</li><br>
	<ul>
	<li><b>mcs/none</b> is the default and disables MCS labels and functionality.
	</li><br>
	<li><b>mcs/account</b> MCS labels can only take a value equal to the job's
	--account. <b>NOTE</b>: this option requires enabling of accounting.
	</li><br>
	<li><b>mcs/group</b> MCS labels can only take a value equal to the job's user
	group.
	</li><br>
	<li><b>mcs/user</b> MCS labels can only take a value equal to the username of
	the job's --uid.
	</li><br>
	<li><b>mcs/label</b> MCS labels are arbitrary strings. Use of a job_submit
	plugin is recommended to manage job submissions to these labels, as no
	enforcement is provided otherwise.
	</li><br>
	</ul>
	</ul>

	<p>MCS_labels on jobs can be displayed through 'squeue' with the format option
	<b>mcslabel</b> or through 'scontrol show job'. Nodes also acquire a MCS_label,
	which is inherited from the allocated job's MCS_label. The node's label can be
	displayed with 'scontrol show nodes'. The 'sview' command can also be used to
	see these MCS_labels.</p>

	<p>MCS_labels on jobs can be updated through 'scontrol update job' with the
	field option <b>MCSLabel</b>. Only PENDING jobs can be modified.

	<p>Users may either request a particular category label for a job (through the
	<b>--mcs-label</b> option), or use the default value generated by the specific
	MCS plugin implementation. So labels can be configured to be enforced or set
	on demand, and the specific MCS Plugin is responsible for checking the validity
	of these labels. When enforced, the MCS Plugin implementation will always
	associate a MCS label to a submitted job, either the default value or the one
	requested by the user (if it's considered correct).</p>

	<p>The selection (exclusivity) of nodes can be filtered on MCS labels either
	on demand (ondemand) or always (select) or never (noselect). Users can force
	the filter with the <b>--exclusive=mcs</b> option (except if noselect mode is
	enabled).
	</p>

	<p>Label enforcement, node selection filtering policy, private data based on
	labels and a list of user groups allowed to be mapped to MCS labels can be
	configured through MCSParameters option.</p>

	<ul>
	<li><b>MCSParameters</b> Specifies the options to pass to the specific MCS
	Plugin implementation. Options should satisfy the following expression:
	<br>
	"[ondemand\|enforced][,noselect\|select\|ondemandselect][,privatedata]:[mcs_plugin_parameters]".
	The defaults are "ondemand,ondemandselect" and no privatedata.
	</li>
	<br>
	<ul>
	<li><b>ondemand\|enforced</b> set MCS label on jobs either on demand (using
	--mcs-label option) or always</li>
	<li><b>noselect\|select\|ondemandselect</b> select nodes with filter on MCS
	label: never, always or on demand (using --exclusive=mcs option)</li>
	<li><b>privatedata</b> accordingly with PrivateData specific option:<br>
	<ul>
	<li>
	If configured with <i>privatedata</i> and <i>PrivateData=jobs</i>, job
	information is filtered based on their MCS labels.<br>
	</li>
	<li>
	If configured with <i>privatedata</i> and <i>PrivateData=nodes</i>, node
	information is filtered based on their MCS labels.<br>
	</li>
	<li>
	Warning: configuring mcs/label with <i>privatedata</i> disables most filtering
	provided by <i>PrivateData=[jobs\|nodes]</i> as all users are presumed to have
	access to all labels.<br>
	</li>
	</ul>
	<li><b>mcs_plugin_parameters</b> Only mcs/group is currently supporting this
	option. It can be used to specify the list of user groups (separated by the
	'<b>\|</b>' character) that are allowed to be mapped to MCS labels by the
	mcs/group plugin.</li>
	</li>
	</ul>
	</li>
	</ul>

	<p>When enforcing privatedata with mcs/account, users will need to have an
	association with just the relevant Account to see other jobs in that Account.
	If a Partition is included in the association, it will be over-constrained
	and will not allow them to view other jobs in the Account.</p>

	Different requests and configurations lead to different combinations of
	use-cases. The following table is intended to help understand the end user
	the expected behavior (related to exclusivity) for a subset of these use-cases:
	<br>
	<br>

	<table style="page-break-inside: avoid; font-family: Arial,Helvetica,sans-serif;" border="1" bordercolor="#000000" cellpadding="6" cellspacing="0" width="100%">
	<tr>
	<td bgcolor="#e0e0e0">
	Node filtering:
	</td>
	<td bgcolor="#e0e0e0">
	Label enforcement: <b>ondemand</b><br>
	(MCS_label set only if requested.)
	</td>
	<td bgcolor="#e0e0e0">
	Label enforcement: <b>enforced</b><br>
	(MCS_label is mandatory.)
	</td>
	</tr>
	<tr>
	<td bgcolor="#e0e0e0">
	<b>noselect</b>
	</td>
	<td>
	No filter on nodes even if --exclusive=mcs requested.
	</td>
	<td>
	No filter on nodes even if --exclusive=mcs requested.
	</td>
	</tr>
	<tr>
	<td bgcolor="#e0e0e0">
	<b>select</b>
	</td>
	<td>
	Filter on nodes <b>only</b> if job MCS_label is set.
	</td>
	<td>
	Always filter on nodes.
	</td>
	</tr>
	<tr>
	<td bgcolor="#e0e0e0">
	<b>ondemandselect</b>
	</td>
	<td>
	Filter on nodes <b>only</b> if --exclusive=mcs.
	</td>
	<td>
	Filter on nodes <b>only</b> if --exclusive=mcs.
	</td>
	</tr>
	</table>

	<h2 id="examples">Some examples<a class="slurm_link" href="#examples"></a></h2>

	<p>slurm.conf:
	<pre>
	MCSPlugin=mcs/account
	MCSParameters=enforced,select,privatedata
	</pre>

	or

	<p>
	<pre>
	MCSPlugin=mcs/group
	MCSParameters=ondemand,noselect:groupA\|groupB\|groupC
	</pre>

	or

	<p>
	<pre>
	MCSPlugin=mcs/user
	MCSParameters=enforced,select,privatedata
	</pre>

	or

	<p>
	<pre>
	MCSPlugin=mcs/label
	MCSParameters=ondemand,select
	</pre>

	<h2 id="view_params">View MCS parameters
	<a class="slurm_link" href="#view_params"></a>
	</h2>

	<p>Configured MCS parameters can be viewed using the <i>scontrol</i> command.
	<pre>
	scontrol show config \|grep MCS
	MCSPlugin = mcs/group
	MCSParameters = ondemand,noselect:groupA\|groupB\|groupC
	</pre>

	<h3 id="add_label">Put an mcs_label on a job using salloc, sbatch or srun
	<a class="slurm_link" href="#add_label"></a>
	</h3>
	<pre>
	srun -n10 -t 1000 --mcs-label=groupB ./job &
	</pre>

	<h3 id="exclusive">Put an mcs_label on a job with exclusivity
	<a class="slurm_link" href="#exclusive"></a>
	</h3>
	<pre>
	srun -n10 -t 1000 --mcs-label=groupB --exclusive=mcs ./job &
	</pre>

	<h3 id="change_label">
	Put a different mcs_label on a job with the mcs/account plugin
	<a class="slurm_link" href="#change_label"></a>
	</h3>
	<pre>
	srun -n10 -t 1000 -A another_account_than_default ./job &
	</pre>

	<h2 id="view_usage">View MCS parameters in usage
	<a class="slurm_link" href="#view_usage"></a>
	</h2>

	<p>The <i>squeue</i> command can also display the MCS with mcslabel
	format option
	<pre>
	squeue -O jobid,username,mcslabel
	JOBID USER MCSLABEL
	2 foo groupA
	3 bar groupB
	</pre>

	<p>The <i>scontrol</i> command will now display the MCS_label
	<pre>
	scontrol show nodes
	NodeName=node0001 Arch=x86_64 CoresPerSocket=4
	CPUAlloc=0 CPUTot=8 CPULoad=0.01 Features=(null)
	Gres=(null)
	NodeAddr=noder0001 NodeHostName=node0001 Version=15.08
	OS=Linux RealMemory=23780 AllocMem=0 Sockets=2 Boards=1
	State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=40 Owner=N/A MCS_label=groupA
	BootTime=2015-08-05T10:14:41 SlurmdStartTime=2015-09-07T13:42:28
	CurrentWatts=0 AveWatts=0
	</pre>

	<p style="text-align:center;">Last modified 20 January 2025</p>

	<!--#include virtual="footer.txt"-->