doc/html/dist_plane.shtml

<!--#include virtual="header.txt"-->

<h1><a name="top">Plane distribution: <i>-m plane=plane_size</i></a></h1>

<p>The plane distribution allocates tasks in blocks of size 
plane_size in a round-robin fashion across allocated nodes.

<p>To specify the plane distribution add to the srun command line
<i>--distribution=plane=plane_size</i> or <i>-m plane=plane_size</i> 
where <i>plane_size</i> is the requested plane/block size.

<h2>Examples of plane distributions</h2>

<p>In the examples below we assume we have 21 tasks and that the
task list is: 0, 1, 2, 3, 4, ..., 19, 20.

<p>On <u>One (1)</u> node: <i>srun -N 1-1 -n 21 -m plane=4 <...></i>.

<p>The distribution results in a plane distribution with plane_size 21.
Even thought the user specified a plane_size of 4 the final plane
distribution results in a plane_size of 21.

<p>
<center>
<img src="plane_ex1.gif">
<p>Figure 1: Process layout for <i>srun -N 1-1 -n 21 -m plane=4 <...></i>
</center>

<p>On <u>four (4)</u> nodes: <i>srun -N 4-4 -n 21 -m plane=4 <...></i>.

<p>The plane distribution with a plane_size of 4 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex2.gif">
<p>Figure 2: Process layout for <i>srun -N 4-4 -n 21 -m plane=4 <...> </i>
</center>

<p>On <u>four (4)</u> nodes: <i>srun -N 4-4 -n 21 -m plane=2 <...>
</i>. 

<p>The plane distribution with a plane_size of 2 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex3.gif">
<p>Figure 3: Process layout for <i>srun -N 4-4 -n 21 -m plane=2 <...></i>
</center>

<p class="footer"><a href="#top">top</a></p>

<h2>Assumptions and Limitations</h2>

<p>SLURM assumes that the number of tasks divided by the plane_size is
greater or equal to the number of nodes specified.

<p>In other words if the following is true SLURM will generate an
error message:
<pre>
if ((n/plane_size < min_nodes) ((N-1)*plane_size >= n))
            generate an error message: Too few processes for the
	           requested {plane, node} distribution.  

where min_nodes is the minimum number of nodes requested for the job.
</pre>

<p class="footer"><a href="#top">top</a></p>

<h2>Plane distribution and task affinity</h2>

<p>The concept behind this distribution is to divide the clusters into
planes. Each plane includes a number of the lowest level of logical
processors (CPU, cores, threads depending on the architecture) on each
node. We then schedule within each plane first and then across planes.

<p>We ensure that the processes are located correctly by setting the
process affinity to the specified/appropriate logical processor. Process
affinity is available in SLURM when the task affinity plug-in is
enabled.

<p>On a dual-processor node with quad-core processors (see figure 4)
the plane distribution results in: 

<ul>
<li>One plane if the plane_size=8. In this case the processors are
scheduled by first filling up the nodes and then scheduled across the
nodes.</li>
<li>Eight planes if the plane_size=1. In this case we would always
schedule across the node first.</li>
</ul>

<p>
<center>
<img src="plane_ex4.gif">
<p>Figure 4: Quad-core dual-processor system
</center>

<p>In a multi-core/hyper-threaded environment, two planes would
provide better locality but potentially more contention for other
resources. 

<p>On the other hand, four planes (scheduling across processors) would
minimize contention for cache and memory.

<p class="footer"><a href="#top">top</a></p>

<h3>Examples of plane distributions with process
affinity enabled</h3>

<p>In the examples below we assume we have 21 tasks and that the
task list is: 0, 1, 2, 3, 4, ..., 19, 20.

<p>On <u>One (1)</u> node: 
<i>srun -N 1-1 -n 21 -m plane=4 --cpu_bind=core <...></i>.
Even thought the user specified a plane_size of 4 the final plane
distribution results in a plane distribution with plane_size=8.

<p>
<center>
<img src="plane_ex5.gif">
<p>Figure 5: Process layout for 
<i>srun -N 1-1 -n 21 -m plane=4 --cpu_bind=core <...></i>.
</center>

<p>On <u>four (4)</u> nodes: 
<i>srun -N 4-4 -n 21 -m plane=4 --cpu_bind=core <...></i>. 
The plane distribution with a plane_size of 4 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex6.gif" width=600>
<p>Figure 6: Process layout for 
<i>srun -N 4-4 -n 21 -m plane=4 --cpu_bind=core <...></i>.
</center>

<p>On <u>four (4)</u> nodes: 
<i>srun -N 4-4 -n 21 -m plane=2 --cpu_bind=core <...>
</i>. The plane distribution with a plane_size of 2 results in the
following allocation of the task ids:

<p>
<center>
<img src="plane_ex7.gif" width=600>
<p>Figure 7: Process layout for 
<i>srun -N 4-4 -n 21 -m plane=2 --cpu_bind=core <...></i>.
</center>

<p class="footer"><a href="#top">top</a></p>

<p style="text-align:center;">Last modified 1 April 2009</p>

<!--#include virtual="footer.txt"-->