<!--#include virtual="header.txt"-->

<h1>Multi-Cluster Operation</h1>

<p>A cluster is comprised of all the nodes managed by a single slurmctld
daemon.  Slurm offers the ability to target commands to other
clusters instead of, or in addition to, the local cluster on which the
command is invoked.  When this behavior is enabled, users can submit
jobs to one or many clusters and receive status from those remote
clusters.</p>

<p>For example:</p>

<PRE>
juser@dawn> squeue -M dawn,dusk
CLUSTER: dawn
JOBID PARTITION   NAME   USER  ST   TIME NODES BP_LIST(REASON)
76897    pdebug  myJob  juser   R   4:10   128 dawn001[8-15]
76898    pdebug  myJob  juser   R   4:10   128 dawn001[16-23]
16899    pdebug  myJob  juser   R   4:10   128 dawn001[24-31]

CLUSTER: dusk
JOBID PARTITION   NAME   USER  ST   TIME NODES BP_LIST(REASON)
11950    pdebug   aJob  juser   R   4:20   128 dusk000[0-15]
11949    pdebug   aJob  juser   R   5:01   128 dusk000[48-63]
11946    pdebug   aJob  juser   R   6:35   128 dusk000[32-47]
11945    pdebug   aJob  juser   R   6:36   128 dusk000[16-31]
</PRE>

<p>Most of the Slurm client commands offer the "-M, --clusters="
option which provides the ability to communicate to and from a comma
separated list of clusters.</p>

<p>When <b>sbatch</b>, <b>salloc</b> or <b>srun</b> is invoked with a cluster
list, Slurm will immediately submit the job to the cluster that offers the
earliest start time subject its queue of pending and running jobs.  Slurm will
make no subsequent effort to migrate the job to a different cluster (from the
list) whose resources become available when running jobs finish before their
scheduled end times.</p>

<p><b>NOTE</b>: In order for <b>salloc</b> or <b>srun</b> to work with the "-M,
--clusters" option in a multi-cluster environment, the compute nodes must be
accessible to and from the submission host.</p>

<h2 id="multi_cluster">Multi-Cluster Configuration
<a class="slurm_link" href="#multi_cluster"></a>
</h2>
<p>The multi-cluster functionality requires the use of the SlurmDBD.
The AccountingStorageType in the slurm.conf file must be set to the
accounting_storage/slurmdbd plugin and the MUNGE or authentication
keys must be installed to allow each cluster to communicate with the
SlurmDBD.  Note that MUNGE can be configured to use different keys for
communications within a cluster and across clusters if desired.
See <a href="accounting.html">accounting</a> for details.</p>

<p>Once configured, Slurm commands specifying the "-M, --clusters="
option will become active for all of the clusters listed by the
<b>"sacctmgr show clusters"</b> command.</p>

<p>
See also the <a href="federation.html">Slurm Federated Scheduling Guide<a/>.
</p>

<!--#include virtual="footer.txt"-->