doc/html/namespace.shtml

<!--#include virtual="header.txt"-->

<h1>Namespace Plugins</h1>

<h2 id="overview">Overview<a class="slurm_link" href="#overview"></a></h2>

<p>A namespace plugin can be enabled to provide private, temporary file system
space for each job.</p>

<p>When enabled on the cluster, a filesystem namespace will be created for each
job with a unique, private instance of /tmp and /dev/shm for the job to use.
These directories can be changed with the <b>Dirs=</b> option in the
plugin-specific configuration file. The contents of these directories will be
removed at job termination. Additionally, the <b>namespace/linux</b> plugin can
be configured to create new IPC, PID, and user namespaces.</p>

<h2 id="installation">Installation
<a class="slurm_link" href="#installation"></a>
</h2>

<p>These plugins are built and installed as part of the default build, no extra
installation steps are required.</p>

<h2 id="setup">Setup<a class="slurm_link" href="#setup"></a></h2>

<p>Slurm must be configured to load the namespace plugin by adding
<b>PrologFlags=contain</b> and setting <b>NamespaceType</b> to the desired
plugin in slurm.conf. Additional configuration must be done in the
plugin-specific configuration file file, which should be placed in the same
directory as slurm.conf.</p>

<p>If using <b>pam_slurm_adopt</b>, note the
<a href="pam_slurm_adopt.html#join_container">join_container</a> option that
controls whether an adopted SSH session will try to join the job's namespace.</p>

<h3 id="linux_setup">namespace/linux plugin
<a class="slurm_link" href="#linux_setup"></a>
</h3>

<p>The <b>namespace/linux</b> plugin (added 25.11) uses the configuration file
<a href="namespace.yaml.html">namespace.yaml</a>. This plugin can be configured
to create IPC, PID, and user namespaces in addition to a temporary filesystem
namespace. Namespaces can be configured for all nodes or for a subset of nodes.
</p>

<p>By default the plugin creates private bind mounts for <b>/tmp</b> and
<b>/dev/shm</b>. The <b>dirs</b> option can be used to change the list of
directories. For more control, the <b>dir_confs</b> option provides
per-directory configuration including per-directory backing storage, mount
options, tmpfs support, and path substitutions.</p>

<p>As an example, if all nodes will be configured the same way, you could put
the following in your namespace.yaml:</p>

<pre>
defaults:
  auto_base_path: true
  base_path: "/var/nvme/storage"
</pre>

<p><b>Note</b> the following important details with this plugin:</p>

<ul>
<li>This plugin requires <b>cgroup/v2</b> to operate correctly.</li>
<li>When using user namespaces, bpf token support (added in kernel 6.9) is
  required to use <a href="cgroup.conf.html#OPT_ConstrainDevices">
  ConstrainDevices</a> in <b>cgroup.conf</b>.</li>
</ul>

<h3 id="tmpfs_setup">namespace/tmpfs plugin
<a class="slurm_link" href="#tmpfs_setup"></a>
</h3>

<p>The <b>namespace/tmpfs</b> plugin (formerly job_container/tmpfs) uses the
configuration file <a href="job_container.conf.html">job_container.conf</a>.
Namespaces can be configured for all nodes, or for a subset of nodes. As an
example, if all nodes will be configured the same way, you could put the
following in your job_container.conf:</p>

<pre>
AutoBasePath=true
BasePath=/var/nvme/storage
</pre>

<h2 id="initial_testing">Initial Testing
<a class="slurm_link" href="#initial_testing"></a>
</h2>

<p>An easy way to verify that the container is working is to run a job and
ensure that the /tmp directory is empty (since it normally has some other
files) and is owned by the user that submitted the job.</p>

<pre>
tim@slurm-ctld:~$ srun ls -al /tmp
total 8
drwx------  2 tim    root 4096 Feb 10 17:14 .
drwxr-xr-x 21 root   root 4096 Nov 15 08:46 ..
</pre>

<p>While a job is running, root should be able to confirm that
<code>/$BasePath/$JobID/_tmp</code> exists and is empty. This directory is bind
mounted into the job. <code>/$BasePath/$JobID</code> should be owned by root,
and is not intended to be accessible to the user.</p>

<p>You can also verify the mount location is correct:</p>

<pre>
parallels@linux_vb:~$ echo $SLURM_JOB_ID
7
parallels@linux_vb:~$ findmnt -o+PROPAGATION | grep /tmp
└─/tmp  /dev/sda1[/storage/7/.7] ext4  rw,relatime,errors=remount-ro,data=ordered   private
</pre>

<p>In the example above, <b>base_path</b> points to <b>/storage</b>, so job ID 7
mounts <b>/tmp</b> on <b>/storage/7/.7</b>. When a user from inside a job tries
to look up mounts, they can see that their <b>/tmp</b> is mounted, but they are
prevented from mistakenly accessing the backing directory directly.</p>

<p>Additionally, when the <b>Linux</b> plugin is in use, you can confirm that a
PID namespace is in effect by running a job and running <code>ps</code>.
The only visible PIDs should be related to the job and PID 1 will be named
<code>slurmstepd: [${job_id}.namespace]</code>.</p>

<h2 id="spank">SPANK<a class="slurm_link" href="#spank"></a></h2>

<p>This plugin interfaces with the SPANK API, and automatically joins the job's
namespace in the following functions:</p>

<ul>
<li>spank_task_init_privileged()</li>
<li>spank_task_init()</li>
</ul>

<p>Mount namespace construction also happens before the job's SPANK environment
is set up. Hence all SPANK-related job steps will view only the private /tmp the
plugin creates.</p>

<p>In addition to the job itself, the <b>TaskProlog</b> will also be executed
inside the container.</p>

<!--#include virtual="footer.txt"-->