doc/html/dist_plane.shtml

<!--#include virtual="header.txt"-->

<h1><a name="top">Plane distribution: <i>-m plane=plane_size</i></a></h1>

<p>The plane distribution allocates tasks in blocks of size
<i>plane_size</i> in a round-robin fashion across allocated nodes.

<p>To specify the plane distribution add to the srun command line
<i>--distribution=plane=plane_size</i> or <i>-m plane=plane_size</i>
where <i>plane_size</i> is the requested plane/block size.

<h2 id="Examples">Examples of plane distributions
<a class="slurm_link" href="#Examples"></a>
</h2>

<p>In the examples below we assume we have 21 tasks and that the
task list is: 0, 1, 2, 3, 4, ..., 19, 20.

<p>On <u>one (1)</u> node: <i>srun -N 1-1 -n 21 -m plane=4 <...></i>.

<p>Even though the user specified a <i>plane_size</i> of 4 the final plane
distribution results in a <i>plane_size</i> of 21, since all the tasks landed
on one node.

<p>
<center>
<img src="plane_ex1.gif">
<p>Figure 1: Process layout for <i>srun -N 1-1 -n 21 -m plane=4 <...></i>
</center>
<br>
<p>On <u>four (4)</u> nodes: <i>srun -N 4-4 -n 21 -m plane=4 <...></i>.

<p>The plane distribution with a <i>plane_size</i> of 4 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex2.gif">
<p>Figure 2: Process layout for <i>srun -N 4-4 -n 21 -m plane=4 <...> </i>
</center>
<br>
<p>On <u>four (4)</u> nodes: <i>srun -N 4-4 -n 21 -m plane=2 <...>
</i>.

<p>The plane distribution with a <i>plane_size</i> of 2 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex3.gif">
<p>Figure 3: Process layout for <i>srun -N 4-4 -n 21 -m plane=2 <...></i>
</center>


<h2 id="Distribution">Plane distribution and task affinity
<a class="slurm_link" href="#Distribution"></a>
</h2>

<p>The concept behind this distribution is to divide the clusters into
planes. Each plane includes a number of the lowest level of logical
processors (CPU, cores, threads depending on the architecture) on each
node. We then schedule within each plane first and then across planes.

<p>We ensure that the processes are located correctly by setting the
process affinity to the specified/appropriate logical processor. Process
affinity is available in Slurm when the task/affinity plug-in is
enabled.

<p>On a dual-processor node with quad-core processors (see figure 4)
the plane distribution results in:

<ul>
<li>One plane if the <i>plane_size=8</i>. In this case the processors are
scheduled by first filling up the nodes and then scheduled across the
nodes.</li>
<li>Eight planes if the <i>plane_size=1</i>. In this case we would always
schedule across the node first.</li>
</ul>

<p>
<center>
<img src="plane_ex4.gif">
<p>Figure 4: Quad-core dual-processor system
</center>
<br>
<p>In a multi-core/hyper-threaded environment, two planes would
provide better locality but potentially more contention for other
resources.

<p>On the other hand, four planes (scheduling across processors) would
minimize contention for cache and memory.


<h2 id="Affinity">Examples of plane distributions with process affinity enabled
<a class="slurm_link" href="#Affinity"></a>
</h2>

<p>In the examples below we assume we have 21 tasks and that the
task list is: 0, 1, 2, 3, 4, ..., 19, 20.

<p>On <u>one (1)</u> node:
<i>srun -N 1-1 -n 21 -m plane=4 --cpu-bind=core <...></i>.
Even though the user specified a <i>plane_size</i> of 4 the final plane
distribution results in a plane distribution with <i>plane_size=8</i>.

<p>
<center>
<img src="plane_ex5.gif">
<p>Figure 5: Process layout for
<i>srun -N 1-1 -n 21 -m plane=4 --cpu-bind=core <...></i>.
</center>
<br>
<p>On <u>four (4)</u> nodes:
<i>srun -N 4-4 -n 21 -m plane=4 --cpu-bind=core <...></i>.
The plane distribution with a <i>plane_size</i> of 4 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex6.gif" width=600>
<p>Figure 6: Process layout for
<i>srun -N 4-4 -n 21 -m plane=4 --cpu-bind=core <...></i>.
</center>
<br>
<p>On <u>four (4)</u> nodes:
<i>srun -N 4-4 -n 21 -m plane=2 --cpu-bind=core <...>
</i>. The plane distribution with a <i>plane_size</i> of 2 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex7.gif" width=600>
<p>Figure 7: Process layout for
<i>srun -N 4-4 -n 21 -m plane=2 --cpu-bind=core <...></i>.
</center>
<br>

<p style="text-align:center;">Last modified 22 April 2021</p>

<!--#include virtual="footer.txt"-->