doc/html/upgrades.shtml

<!--#include virtual="header.txt"-->

<h1>Upgrade Guide</h1>

<p>Slurm supports in-place upgrades between certain versions. This page provides
important details about the steps necessary to perform an upgrade and the
potential complications to prepare for.</p>

<p>See also <a href="quickstart_admin.html">Quick Start Administrator Guide</a></p>

<h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>
<ul>
<li><a href="#release_cycle">Release Cycle</a>
<ul>
<li><a href="#compatibility_window">Compatibility Window</a></li>
<li><a href="#epel_repository">EPEL Repository</a></li>
<li><a href="#prerelease">Pre-Release Versions</a></li>
</ul></li>
<li><a href="#revert">Reverting an Upgrade</a></li>
<li><a href="#minor_upgrades">Minor Upgrades</a></li>
<li><a href="#procedure">Upgrade Procedure</a>
<ul>
<li><a href="#preparation">Preparation</a></li>
<li><a href="#backups">Create Backups</a></li>
<li><a href="#slurmdbd">slurmdbd (Accounting)</a>
<ul>
<li><a href="#db_server">Database Server</a></li>
</ul></li>
<li><a href="#slurmctld">slurmctld (Controller)</a></li>
<li><a href="#slurmd">slurmd (Compute Nodes)</a></li>
<li><a href="#other_commands">Other Slurm Commands</a></li>
<li><a href="#custom_plugins">Customized Slurm Plugins</a></li>
<li><a href="#stepmgr_hetjobs">Step Manager and Heterogeneous Jobs (26.11+)</a></li>
</ul></li>
<li><a href="#seamless_upgrades">Seamless Upgrades</a></li>
</ul>

<h2 id="release_cycle">Release Cycle
<a class="slurm_link" href="#release_cycle"></a></h2>

<p>The Slurm version number contains three period-separated numbers that
represent both the major Slurm release and maintenance release level.
For example, Slurm 23.11.4:</p>

<ul>
<li><b>23.11</b> = major release
<ul>
<li>This matches the year and month of initial release (November 2023)</li>
<li>Major releases may contain changes to RPCs (remote procedure calls),
	state files, configuration options, and core functionality</li>
</ul></li>
<li><b>.4</b> = maintenance version
<ul>
<li>Maintenance releases may contain bug fixes and performance improvements</li>
</ul></li>
</ul>

<p>Since version 24.05, Slurm has followed a 6-month release cycle for major
releases, and each release is supported for 18 months. Once a release leaves
its support window, it no longer receives bug fixes or security patches. To
keep your site secure and stable, plan to upgrade to a supported release
before your current one reaches end of support. Upgrading regularly also
gives you the latest performance improvements and features.</p>

<h3 id="compatibility_window">Compatibility Window
<a class="slurm_link" href="#compatibility_window"></a></h3>

<p>Starting with version 24.11, Slurm is compatible with the <b>three
previous major releases</b>, which matches the 18-month support window. For
example, slurmdbd 25.11.x can accept messages from slurmctld daemons and
commands running version 25.05.x, 24.11.x, or 24.05.x. It can also update
records in the database that were written by a slurmdbd running any of these
versions.</p>

<table class="tlist centered">
<tbody>
<tr>
<td><strong>Slurm Release</strong></td>
<td><strong>End of Support</strong><br>(duration)</td>
<td><strong>Compatible Prior Versions</strong></td>
</tr>
<tr>
<td>23.11</td>
<td>May 2025 (18 months)</td>
<td>23.02, 22.05</td>
</tr>
<tr>
<td>24.05</td>
<td>November 2025 (18 months)</td>
<td>23.11, 23.02</td>
</tr>
<tr>
<td>24.11</td>
<td>May 2026 (18 months)</td>
<td>24.05, 23.11, 23.02</td>
</tr>
<tr>
<td>25.05</td>
<td>November 2026 (18 months)</td>
<td>24.11, 24.05, 23.11</td>
</tr>
<tr>
<td>25.11</td>
<td>May 2027 (18 months)</td>
<td>25.05, 24.11, 24.05</td>
</tr>
<tr>
<td>26.05</td>
<td>November 2027 (18 months)</td>
<td>25.11, 25.05, 24.11</td>
</tr>
<tr>
<td>26.11</td>
<td>May 2028 (18 months)</td>
<td>26.05, 25.11, 25.05</td>
</tr>
<tr>
<td>27.05</td>
<td>November 2028 (18 months)</td>
<td>26.11, 26.05, 25.11</td>
</tr>
</tbody>
</table>
<br>

<p>Upgrades from incompatible versions will fail immediately upon startup.
It is required to perform upgrades from incompatible prior versions in steps,
going to newer versions compatible with the current running version. It may
take several steps to upgrade to a current release of Slurm. For example,
instead of upgrading directly from Slurm 20.11 to 23.11, first upgrade all
systems to Slurm 22.05 and verify functionality, then proceed to upgrade to
23.11. This ensures that each upgrade performed is tested and can be supported
by SchedMD. Compatibility requirements apply to running jobs and upgrading
outside of their compatibility window will result in the jobs being killed and
job accounting being lost.</p>

<h3 id="epel_repository">EPEL Repository
<a class="slurm_link" href="#epel_repository"></a></h3>

<p>In the beginning of 2021, a version of Slurm was added to the
EPEL repository. This version is not provided by or supported by SchedMD, and is
not currently supported for customer use. Unfortunately, this inclusion could
cause Slurm to be updated to a newer version outside of a planned maintenance
period or result in conflicting packages. In order to prevent Slurm from being
changed and broken unintentionally, we recommend you modify the EPEL Repository
configuration to exclude all Slurm packages from automatic updates.</p>

<p>Add the following under the <code>[epel]</code>
section of /etc/yum.repos.d/epel.repo:
<pre>exclude=slurm*</pre></p>

<h3 id="prerelease">Pre-Release Versions
<a class="slurm_link" href="#prerelease"></a></h3>

<p>When installing pre-release versions (e.g., 24.05.0rc1 or
<a href="https://github.com/SchedMD/slurm">master branch</a>), you should prepare
for unexpected crashes, bugs, and loss of state information. SchedMD aims to
use the NEWS file to indicate cases in which state information will be lost with
pre-release versions. However, these pre-release versions receive <b>limited
testing</b> and are not intended for production clusters. Sites are encouraged
to actively run pre-release versions on test machines before each major release.
</p>

<h2 id="revert">Reverting an Upgrade
<a class="slurm_link" href="#revert"></a></h2>

<p>Reverting an upgrade (or downgrading) is <b>not supported</b> once any of the
Slurm daemons have been started. When starting up after an upgrade, the Slurm
daemons (slurmctld, slurmdbd, and slurmd) will update their relevant state
files and databases to the structure used in the new version. If you revert to
an older version, the relevant Slurm daemon will not recognize the new state
file or database, resulting in loss or corruption of state information or job
accounting. The Slurm daemons will likely refuse to start unless configured to
start with the risk of possible data loss.</p>

<p>By using recovery tools, like comprehensive file backups, disk images, and
snapshots, it may be possible to revert components to the pre-upgrade state.
In particular, restoring the contents of <i>StateSaveLocation</i> (as defined in
<i>slurm.conf</i>) and (if configured) the accounting database will be required
if you wish to revert an upgrade. Reverting an upgrade will wipe out anything
that happened after the backups were created.</p>

<h2 id="minor_upgrades">Minor Upgrades
<a class="slurm_link" href="#minor_upgrades"></a></h2>

<p>When upgrading to a newer minor maintenance release (as
<a href="#release_cycle">defined above</a>), we recommend following the same
upgrade procedure as with major releases. You will find that the process takes
less time, and is more accommodating of mixed versions and in-place
downgrades. However, you should always have current backups to solidify your
recovery options.</p>

<h2 id="procedure">Upgrade Procedure
<a class="slurm_link" href="#procedure"></a></h2>

<p>The upgrades procedure can be summarized as follows. Note the specific order
in which the daemons should be upgraded:</p>

<ol>
<li><a href="#preparation">Prepare cluster for the upgrade</a></li>
<li><a href="#backups">Create backups</a></li>
<li>Upgrade <a href="#slurmdbd">slurmdbd</a></li>
<li>Upgrade <a href="#slurmctld">slurmctld</a></li>
<li>Upgrade <a href="#slurmd">slurmd</a> (preferably with slurmctld)</li>
<li>Upgrade <a href="#other_commands">login nodes and client commands</a></li>
<li>Recompile/upgrade <a href="#custom_plugins">customized Slurm plugins</a></li>
<li>Test key functionality</li>
<li>Archive backup data</li>
</ol>

<p>Before considering the upgrade complete, wait for all jobs that were already
running to finish. Any jobs started before the <b>slurmd</b> system was upgraded
will be running with the old version of <b>slurmstepd</b>, so starting another
upgrade or trying to use new features in the new version may cause problems.</p>

<p><b>NOTE</b>: If RPM/DEB packages are used, all packages present on each
system must be upgraded together instead of piecewise. This is because the
packages that contain Slurm daemons and client commands depend on the general
<b>slurm</b> package. Avoid using low-level package managers like <code>rpm</code>
or <code>dpkg</code> as they may not properly enforce these dependencies.
After upgrading, daemons should be started in the order listed above.</p>

<h3 id="preparation">Preparation
<a class="slurm_link" href="#preparation"></a></h3>

<h4 id="release_notes">RELEASE_NOTES and CHANGELOG
<a class="slurm_link" href="#release_notes"></a></h4>

<p>Review relevant release notes in the <b>RELEASE_NOTES.md</b> file in root of
Slurm source directory for the target release and any major versions between
what you're currently running and the target you are upgrading to. Pay
particular attention to any entries in which items are <b>removed</b> or
<b>changed</b>. These are particularly likely to require specific attention or
changes during the upgrade. Also look for changes in optional slurm components
that you are using. You may also notice new items added to Slurm that you wish
to start using after the upgrade.</p>

<p>Release notes for the latest major version are
available <a href="release_notes.html">here</a> or on GitHub
(<a href="https://github.com/SchedMD/slurm/blob/master/RELEASE_NOTES.md">RELEASE_NOTES.md</a>).
Release notes for other versions can be found in the source, which can be viewed
on GitHub by selecting the branch or tag corresponding to the desired version.
(Refer to
<a href="https://github.com/SchedMD/slurm/blob/slurm-24.11/RELEASE_NOTES">
RELEASE_NOTES</a> for Slurm 24.11 and older.) More detailed changes, including
minor release changes, can be found in the <b>CHANGELOG</b> directory (formerly
the NEWS file), but are usually not needed to prepare for upgrades.</p>

<h4 id="config_changes">Configuration Changes
<a class="slurm_link" href="#config_changes"></a></h4>

<p>Always prepare and test configuration changes in a test environment
before upgrading in production. Changes outlined in the release notes will need
to be looked up in the man pages (such as <a href="slurm.conf.html">slurm.conf
</a>) for details and new syntax. Certain options in your configuration files
may need to be changed as features and functionality are improved in every major
Slurm release. Typically, new naming and syntax conventions are introduced
several versions before the old ones are removed, so you may be able to make the
necessary changes before starting the upgrade process.</p>

<p>If <b>SlurmctldParameters=reconfig_on_restart</b> is enabled extra caution
should be taken as the slurmd and sackd daemons could be upgraded earlier than
anticipated. Temporarily disabling that option may be advisable.</p>

<h4 id="downtime">Plan for Downtime
<a class="slurm_link" href="#downtime"></a></h4>

<p>Refer to the expected downtime guidance in the
following sections for each relevant Slurm daemon, particularly the
<a href="#slurmdbd">slurmdbd</a>. Notify affected users of the estimated
downtime for the relevant services and the potential impact on their jobs.
Whenever possible, try to plan upgrades during SchedMD's support hours.
If you encounter an issue outside of these hours there will be a delay before
assistance can be provided.</p>

<h4 id="openapi_changes">OpenAPI Changes
<a class="slurm_link" href="#openapi_changes"></a></h4>

<p>Sites using <code>--json</code> or <code>--yaml</code> arguments with any CLI
commands or running <code>slurmrestd</code> need to check for format
compatibility and data_parser plugin removals before upgrading. The formats for
the values parsed and dumped as JSON and YAML are handled by the data_parser
plugins. Please see the following links for explanations of the changes in each
release and how to account for them while upgrading:</p>

<ul>
<li><a href="rest_clients.html#openapi-changes">OpenAPI Specification (OAS) changes</a></li>
<li><a href="openapi_release_notes.html">OpenAPI release notes</a></li>
</ul>

<h3 id="backups">Create Backups
<a class="slurm_link" href="#backups"></a></h3>

<p><b>Always</b> create full backups to restore all parts of Slurm, including
the Mysql database, before upgrading in the event the upgrade must be reverted.
SchedMD aims to make supported upgrades a seamless process but it is possible
for unexpected issues to arise and <b>irreversibly corrupt</b> all of the data
kept by Slurm. If something like this happens, it will not be possible to
recover any corrupted data and you will be reliant on backed up data.</p>

<p>It is recommended to prepare recovery options (file backups, disk images,
snapshots, database dumps) that will take you back to a known working cluster
state. How backups are taken is specific to how the systems integrator
designed and setup the cluster and procedures are not provided here.</p>

<p>At a minimum, back up the following:
<ul>
<li><b>StateSaveLocation</b> as defined in
<a href="slurm.conf.html#OPT_StateSaveLocation">slurm.conf</a>, or it can be
queried by calling <pre>scontrol show config | grep StateSaveLocation</pre></li>
<li><b>Entire slurm configuration directory</b>, as defined by
<code>configure --sysconfdir=DIR</code> during compilation.
This is usually located in <code>/etc/slurm/</code></li>
<li><b>MySQL database</b> (if slurmdbd is configured). Usually done by calling
<pre>
mysqldump --databases slurm_acct_db &gt; /path/to/offline/storage/backup.sql
</pre>
This assumes that <b>slurmdbd</b> is not running while the dump is running.
<br>If you wish to back it up while <b>slurmdbd</b> is running, you may use the
<code>--single-transaction</code> flag with the <b>following limitations</b>:
<ol>
<li>Database operations may be slower while the dump is running</li>
<li>Restoring this dump will restore the database at the time the dump was
<b>started</b>, losing any changes made during or after the dump</li>
<li>Certain cluster operations may lead to an incorrect or failed dump:
<ul>
<li>Creating a new database</li>
<li>Upgrading an existing database</li>
<li>Adding or Removing a cluster in the slurmdbd</li>
<li><a href="https://slurm.schedmd.com/accounting.html#slurmdbd-archive-purge">
Archiving or Purging</a> accounting data</li>
</ul>
</li>
</ol>
</li>
</ul></p>

<h3 id="slurmdbd">slurmdbd (Accounting)
<a class="slurm_link" href="#slurmdbd"></a></h3>

<p>If <b>slurmdbd</b> is used in your environment, it must be at the same or
higher major release number as the slurmctld daemon(s), and at a close enough
version for <a href="#compatibility_window">compatibility</a>. Thus, when
performing upgrades, it should be upgraded first. When a backup slurmdbd host
is in use, it should be upgraded at the same time as the primary.</p>

<p>Upgrades to the slurmdbd may require significant <b>downtime</b>.
With large accounting databases, the precautionary database dump will take some
time, and the upgraded daemon may be unresponsive for tens of minutes while it
updates the database to the new schema. Sites are encouraged to use the
<a href="slurmdbd.conf.html#OPT_PurgeJobAfter">purge functionality</a> if older
accounting data is not required for normal operations. Purging old records
before attempting to upgrade can significantly decrease outage time.</p>

<p>The non-slurmdbd functionality of the cluster will continue to operate while
the upgrade is in process, provided the activity does not fill up the slurmdbd
Agent queue on the slurmctld node.  While slurmdbd is offline, you should
monitor the memory usage of slurmctld, and the <b>DBD Agent queue size</b>, as
reported by <b>sdiag</b>, to ensure it does not exceed the configured
<b>MaxDBDMsgs</b> in <a href="slurm.conf.html#OPT_MaxDBDMsgs">slurm.conf</a>.
Cli commands <a href="sacct.html">sacct</a> and <a href="sacctmgr.html">
sacctmgr</a> will not work while slurmdbd is down.
<code>slurmrestd</code> queries that include slurmdb in
the URL path will fail while slurmdbd is down.</p>

<p>It is preferred to create a backup of the database after shutting down the
<b>slurmdbd</b> daemon, when the MySQL database is no longer changing. If you
wish to take a backup with <b>mysqldump</b> while the slurmdbd is still
running, you can add <code>--single-transaction</code> to the mysqldump command.
Note that the slurmdbd will continue to execute operations that will not be
contained in the dump, which may cause complications if you need to restore
the database to this state.</p>

<p>The suggested upgrade procedure is as follows:</p>

<ol>
<li>Shutdown the slurmdbd daemon(s) gracefully:
<pre>sacctmgr shutdown</pre>or via systemd:
<pre>systemctl stop slurmdbd</pre> Wait until slurmdbd is fully down before
proceeding or there may be data loss from data that was not fully saved.
<pre>systemctl status slurmdbd</pre>
</li>
<li><a href="#backups">Backup the Slurm database</a></li>
<li>Verify that the innodb_buffer_pool_size in my.cnf is greater than the
default. See the recommendation in the
<a href="accounting.html#slurm-accounting-configuration-before-build">
	accounting page</a>.</li>
<li>Verify that the database server meets the minimum version: MySQL 8.0.21
	or later, or MariaDB 10.2.7 or later. See
	<a href="#db_server">Database Server</a>.</li>
<li>Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file
	(if used). If using <a href="quickstart_admin.html#build_install">
	RPM/DEB	packages</a>, the package manager will take care of these,
	although systemd overrides may prevent the new unit from taking	effect.
	<br>Only upgrade the slurmdbd system(s) at this time; other Slurm
	systems should remain on the old version.</li>
<li>Start the primary slurmdbd daemon.
	<br><b>NOTE</b>: If you typically use systemd, it is recommended to
	initially start the daemon directly as the configured SlurmUser:
	<br><code>sudo -u slurm slurmdbd -D</code>
	<br>When the daemon starts up for the first time after upgrading, it
	will take some extra time to update existing records in the database. If
	it is started with systemd and reaches the configured timeout value, it
	may be killed prematurely potentially causing data loss. After it
	finishes starting up, you can use <code>Ctrl+C</code> to exit, then
	start it normally with systemd.</li>
<li>Start the backup slurmdbd daemon (if applicable).</li>
<li>Validate accounting operation, such as retrieving data through
	<code>sacct</code> or <code>sacctmgr</code>.</li>
</ol>

<h4 id="db_server"><b>Database Server</b>
<a class="slurm_link" href="#db_server"></a></h4>

<p>Slurm requires MySQL 8.0.21 or later, or MariaDB 10.2.7 or later.
The <b>slurmdbd</b> daemon logs a fatal error and exits at startup if the
server it connects to is older than the minimum, so upgrade the database
server <b>before</b> upgrading slurmdbd. See
<a href="platforms.html#database">Platforms</a>.</p>

<p>When upgrading the database server that is used by slurmdbd (e.g., MySQL or
MariaDB), usually no special procedures are required. It is recommended to use a
database server that is supported by the publisher. Database upgrades should be
performed while the slurmdbd is stopped and according to the recommended
procedure for the database used.</p>

<p>When migrating an existing accounting database from MySQL to MariaDB, or
from a MariaDB older than 10.2.1, ensure you are running
<b>slurmdbd 22.05.7</b> or later. These versions gracefully handle changes to
MariaDB default values that can otherwise cause problems for slurmdbd.</p>

<h3 id="slurmctld">slurmctld (Controller)
<a class="slurm_link" href="#slurmctld"></a></h3>

<p>It is preferred to upgrade the slurmctld system(s) at the same time as slurmd
on the compute nodes and other Slurm commands on client machines and login nodes.
The effects of downtime on slurmctld and slurmd daemons are largely the same,
so upgrading them all together minimizes the total duration of these effects.
Rolling upgrades are also possible if the slurmctld is upgraded first. When
multiple slurmctld hosts are used, all should be upgraded simultaneously.</p>

<p>Upgrading the slurmctld involves a brief period of <b>downtime</b> during
which job submissions are not accepted, queued jobs are not scheduled, and
information about completing jobs is held. These functions will resume once
the upgraded controller is started.</p>

<p>The recommended upgrade procedure is below, including optional steps for a
simultaneous upgrade of slurmd systems:</p>

<ol>
<li>Increase configured SlurmdTimeout and SlurmctldTimeout values and
	execute <code>scontrol reconfig</code> for them to take effect.
	<br>The new timeout should be long enough to perform the upgrade using
	your preferred method. If the timeout is reached, nodes may be marked
	DOWN and their jobs killed.</li>
<li>Shutdown the slurmctld daemon(s).</li>
<li>(opt.) Shutdown the slurmd daemons on the compute nodes.</li>
<li>Back up the contents of the configured StateSaveLocation.</li>
<li>Upgrade the slurmctld (and optionally slurmd) daemons and their systemd
	service files (if used).</li>
<li>(opt.) Restart the slurmd daemons on the compute nodes.</li>
<li>Restart the slurmctld daemon(s).</li>
<li>Validate proper operation, such as communication with nodes and a job's
	ability to successfully start and finish.</li>
<li>Restore the preferred SlurmdTimeout and SlurmctldTimeout values and
	execute <code>scontrol reconfig</code> for them to take effect.</li>
</ol>

<h3 id="slurmd">slurmd (Compute Nodes)
<a class="slurm_link" href="#slurmd"></a></h3>

<p>It is preferred to upgrade all slurmd nodes at the same time as the slurmctld.
It is also possible to perform a rolling upgrade by upgrading the slurmd nodes
later in any number of groups. Sites are encouraged to minimize the amount of
time during which mixed versions are used in a cluster.</p>

<p>Upgrades will not interrupt running jobs as long as <b>SlurmdTimeout</b>
is not reached during the process. However, while the slurmd is down for
upgrades, new jobs will not be started and finishing jobs will wait to
report back to the controller until it comes back online.</p>

<p>If you are upgrading the slurmd nodes separately from the controller, the
following procedure can be followed:</p>

<ol>
<li>Increase the configured SlurmdTimeout value and execute
	<code>scontrol reconfig</code> for it to take effect.
	<br>The new timeout should be long enough to perform the upgrade using
	your preferred method. If the timeout is reached, nodes may be marked
	DOWN and their jobs killed.</li>
<li>Shutdown the slurmd daemons on the compute nodes.</li>
<li>Back up the contents of the configured StateSaveLocation.</li>
<li>Upgrade the slurmd daemons and their systemd unit files (if used).</li>
<li>Restart the slurmd daemons.</li>
<li>Validate proper operation, such as communication with the controller and a
	job's ability to successfully start and finish.</li>
<li>Repeat for any other groups of nodes that need to be upgraded.</li>
<li>Restore the preferred SlurmdTimeout value and
execute <code>scontrol reconfig</code> for it to take effect.</li>
</ol>

<h3 id="other_commands">Other Slurm Commands
<a class="slurm_link" href="#other_commands"></a></h3>

<p>Other Slurm commands (including client commands) do not require special
attention when upgrading, except where specifically noted in the release notes.
You should also pay attention to any changes introduced in these additional
components. After core Slurm components have been upgraded, upgrade additional
components along with their systemd unit files (if used) and client commands
using the normal method for your system, then restart any affected daemons.</p>

<h3 id="custom_plugins">Customized Slurm Plugins
<a class="slurm_link" href="#custom_plugins"></a></h3>

<p>Slurm's main public API library (libslurm.so.X.0.0) increases its version
number with every major release, so any application linked against it should be
recompiled after an upgrade. This includes locally developed Slurm plugins.</p>

<p>If you have built your own version of Slurm plugins, besides having to
recompile them, they will likely need modification to support the new version
of Slurm. It is common for plugins to add new functions and function arguments
during major updates. See the RELEASE_NOTES file for details about these
changes.</p>

<p>Slurm's PMI-1 (libpmi.so.0.0.0) and PMI-2 (libpmi2.so.0.0.0) public API
libraries do not change between releases and are meant to be permanently
fixed. This means that linking against either of them will not require you
to recompile the application after a Slurm upgrade, except in the unlikely
event that one of them changes. It is unlikely because these libraries must
be compatible with any other PMI-1 and PMI-2 implementations. If there was a
change, it would be announced in the RELEASE_NOTES and would only happen on
a major release.</p>

<p>As an example, MPI stacks like OpenMPI and MVAPICH2 link against Slurm's
PMI-1 and/or PMI-2 API, but not against our main public API. This means that at
the time of writing this documentation, you don't need to recompile these
stacks after a Slurm upgrade. One known exception is MPICH. When MPICH is
compiled with Slurm support and with the Hydra Process Manager, it will use
the Slurm API to obtain job information. This link means you will need to
recompile the MPICH stack after an upgrade.</p>

<p>One easy way to know if an application requires a recompile is to inspect all
of its ELF files with 'ldd' and grep for 'slurm'. If you see a versioned
'libslurm.so.x.y.z' reference, then the application will likely need to be
recompiled.</p>

<h3 id="stepmgr_hetjobs">Step Manager and Heterogeneous Jobs (26.11+)
<a class="slurm_link" href="#stepmgr_hetjobs"></a></h3>

<p>Starting in Slurm 26.11, the steps of
<a href="heterogeneous_jobs.html">heterogeneous jobs</a> are also managed by
the step manager when it is enabled
(<a href="slurm.conf.html#OPT_enable_stepmgr">
SlurmctldParameters=enable_stepmgr</a>
or <code>--stepmgr</code>). In older versions, heterogeneous jobs always had
their steps managed by slurmctld, regardless of this setting.</p>

<p>Steps of heterogeneous jobs managed by the step manager can only be
launched when both the slurmd on the compute nodes and the <code>srun</code>
that launches them are 26.11 or later. During a rolling upgrade with a 26.11+
slurmctld, such steps fail to launch while compute nodes still run an older
slurmd, or while users still run an older <code>srun</code>. Sites that use
the step manager with heterogeneous jobs should either upgrade slurmctld, all
slurmd daemons and the client commands (including <code>srun</code>) at the
same time, or disable the step manager until all of them have been
upgraded.</p>

<h2 id="seamless_upgrades">Seamless Upgrades
<a class="slurm_link" href="#seamless_upgrades"></a></h2>

<p>In environments where the Slurm build process is customized, it is possible
to install a new version of Slurm to a unique directory and use a symbolic link
to point the directory in your PATH to the version of Slurm you would like to
use. This allows you to install the new version before you are in a maintenance
period as well as easily switch between versions should you need to roll
back for any reason. It also avoids potential problems with library conflicts
that might arise from installing different versions to the same directory.</p>

<!--#include virtual="footer.txt"-->