<!--#include virtual="header.txt"-->

<h1>JSON Web Tokens (JWT) Authentication</h1>

<p>Slurm provides a
<a href="https://datatracker.ietf.org/doc/html/rfc7519">RFC7519</a> compliant
implementation of <a href="https://jwt.io/">JSON Web Tokens (JWT)</a>. This
authentication can be used as an <b>AuthAltType</b>, usually alongside
<b>auth/munge</b> as the <b>AuthType</b>. The only supported communication
direction is from a client connecting to <b>slurmctld</b> and <b>slurmdbd</b>.
This means that certain scenarios (specifically interactive jobs using
<b>srun</b>) are currently not supported for clients with auth/jwt enabled (or
that have SLURM_JWT in their environment).</p>

<h2 id="prerequisites">Prerequisites
<a class="slurm_link" href="#prerequisites"></a>
</h2>
<p>JWT requires <a href="related_software.html#jwt">libjwt</a>.
Both the library and the development headers must be available when Slurm is
compiled.</p>

<h2 id="trust_jwt_create">Full root trust in JWT creation
<a class="slurm_link" href="#trust_jwt_create"></a>
</h2>
<p>The power to create JWTs is the power of root on a cluster. This is a
per-site decision on what/who/when/how to trust. If a given authentication
system can not be fully trusted with the powers of root for the entire cluster,
then an authenticating proxy will need to be used to divide up the trust and
implement the site's specific policies before the requests reach Slurm
(specifically slurmrestd). While possibly inefficient, there is no technical
reason that tiers of authenticating proxies can not be used if there is a lack
of trust but a desire to allow creation of lesser auth tokens. Each site will
need to weight the risks and benefits of which JWTs to trust before implementing
any system. Once a job has been queued, the proxied authentication system will
no longer be involved and the job will run with that user's permissions and
access per Linux/POSIX's ACLs and trusts.</p>

<h2 id="models_trust">Models of trust
<a class="slurm_link" href="#models_trust"></a>
</h2>
<p>There are several ways to handle controlling JWT authentication and access.
Slurm JWT plugin implementation is purposefully simple and will not be able to
support most models of trust needed by sites. There already exists a plethora of
authentication systems, and the expectation is that any site that wants more
complexity than the default offering will use one of those systems instead.</p>

<ul>
<li><a href="#compatibility">External JWT generation</a>
<p>We provide an example python script for generating new JWTs but they are a
standard and most languages have existing tooling for them. This is usually the
easiest route for sites but does require each site to implement the tooling for
their users directly.</p></li>

<li>Authenticating proxy
<p>This is the most versatile option, as any authentication system can be
placed in front of slurmrestd. It requires creating a slurmuser/root token that
can then be used to proxy for any user. There are existing solutions for this
with Nginx and Apache, and probably every other non-trivial proxy. We suggest
choosing the preferred proxy and finding an existing setup guide for
authenticating via that proxy. The proxy will need to have the
<b>X-SLURM-USER-TOKEN</b> and <b>X-SLURM-USER-NAME</b> headers defined.</p>

<p>There is no requirement that an authenticating proxy implement JWT for
clients. This is the primary benefit of authenticating proxies; they can use
any authentication method since they are the trusted point that tells Slurm
which user the request is from. These authentication tokens are only used by
the proxy and are not passed to the job. This is generally not an
issue as once the job is in Slurm, it runs as the Posix user with all of the
inherent trust of that user and it then uses <code>auth/munge</code> or
<code>auth/slurm</code> for everything after that.</p></li>

<li><a href="#external_auth">JWKS</a>
<p>This is like an authentication proxy, as another system is used to create the
tokens, but it skips having the authentication system in front of Slurm by using
signed public keys. This tends to be the preferred solution for sites using
cloud authentication systems, such as:
<ul>
<li><a href="https://aws.amazon.com/blogs/hpc/using-the-slurm-rest-api-to-integrate-with-distributed-architectures-on-aws/">
Amazon Cognito</a></li>
<li><a href="https://www.keycloak.org/">Keycloak</a> - Using keycloak is an
option which doesn't require a cloud auth solution.</li>
</ul>
</p></li>
</ul>

<h2 id="setup">Setup for Standalone Use
<a class="slurm_link" href="#setup"></a>
</h2>
<ol>
<li><a href="related_software.html#jwt">
    Configure and build Slurm with JWT support</a></li>
<li>Add the same JWT key to both the controller and slurmdbd (if used). For the
controller only, it is recommended to put the JWT key in the StateSaveLocation.
For example, using /var/spool/slurm/statesave/:
<pre>
dd if=/dev/random of=/var/spool/slurm/statesave/jwt_hs256.key bs=32 count=1
chown slurm:slurm /var/spool/slurm/statesave/jwt_hs256.key
chmod 0600 /var/spool/slurm/statesave/jwt_hs256.key
chown slurm:slurm /var/spool/slurm/statesave
chmod 0755 /var/spool/slurm/statesave
</pre>
The key does not have to be in the StateSaveLocation, but that is a convenient
location if you have multiple controllers since it is shared between them.
The key should not be placed in a directory where non-admin users might be
able to access it.
The key file should be owned by <b>SlurmUser</b> or <b>root</b>, with
recommended permissions of 0400. The file must not be accessible by 'other'.
</li>
<li>In both slurm.conf and slurmdbd.conf, add JWT as an alternative
authentication type:
<pre>
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/var/spool/slurm/statesave/jwt_hs256.key
</pre>
</li>
<li>Restart slurmctld</li>
<li>Create tokens for users as desired:
<pre>
scontrol token username=$USER
</pre>
An optional <b>lifespan=$LIFESPAN</b> option can be used to change the token
lifespan from the default 1800 seconds. The root account, or <b>SlurmUser</b>
account can be used to generate tokens for any user. Alternatively, a user
may use the command to generate tokens for themselves by simply calling
<pre>
scontrol token
</pre>
Note that administrators can prevent users from generating tokens by setting
the following parameter in slurm.conf:
<pre>
AuthAltParameters=disable_token_creation
</pre>
This functionality is provided to allow sites to control when and how users are
provided tokens along with controlling the token lifespans.
</li>
<li>Export the <b>SLURM_JWT</b> environment variable before calling any Slurm
command.</li>
<li>Export the <b>SLURM_JWT=daemon</b> environment variable before starting
the slurmrestd daemon to activate <i>AuthAltTypes=auth/jwt</i> as the primary
authentication mechanism.</li>
</ol>

<h2 id="external_auth">
External Authentication Integration with JWKS and RS256 Tokens
<a class="slurm_link" href="#external_auth"></a>
</h2>
<p>Starting with the 21.08 release, Slurm can support RS256 tokens such as
those generated by
<a href="https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html">Amazon Cognito</a>,
<a href="https://azure.github.io/azure-workload-identity/docs/installation/self-managed-clusters/oidc-issuer/jwks.html">Azure AD</a>, or
<a href="https://www.keycloak.org/docs/latest/securing_apps/#_client_authentication_adapter">Keycloak</a>.
</p>

<p>To enable Slurm's RS256 token support, an appropriate JWKS file must be
downloaded and configured as such:
<pre>
AuthAltTypes=auth/jwt
AuthAltParameters=jwks=/var/spool/slurm/statesave/jwks.json
</pre>
</p>

<p>The jwks file should be owned by <b>SlurmUser</b> or <b>root</b>, must be
readable by <b>SlurmUser</b>, with recommended permissions of 0400.
The file must not be writable by 'other'.</p>
<p>Note that, by default, the built-in ability to generate HS256 tokens will
be disabled when JWKS support is enabled. It can be re-enabled by explicitly
configuring the <b>jwt_key=</b> option alongside <b>jwks=</b>.
</p>
<p>Note: Slurm ignores the <b>x5c</b> and <b>x5t</b> fields and does not
attempt to verify the certificate chain if presented in the JWKS file. JWTs are
only verified against RSA 256 bit keys provided via <b>e</b> and
<b>n</b> fields.
</p>
<p>JWKS has signing keys that receive trust by being placed in the
jwks.json. Those trusted keys can then create new tokens (which are JWTs) for
any user by signing them. JWKS does not support adding keys for individual
users but only for adding trusted signing keys.
</p>
<p>JWT and JWKS can coexist in Slurm. Slurm will auto-disable JWT when
JWKS is configured as a safety mechanism, to avoid accidentally having both
enabled at the same time.
</p>

<h3 id="keycloak">Keycloak
<a class="slurm_link" href="#keycloak"></a></h3>

<p><a href="https://www.keycloak.org/">KeyCloak</a> can be used to manage
authentication tokens through JWT or JWKS. Certain items should be customized
when using it with Slurm. Refer to the
<a href="https://www.keycloak.org/getting-started/getting-started-docker">
official documentation</a> for initial setup. Or, if you need to automate the
initial setup, you can implement the following customizations:</p>

<ol>
<li><code>startup.sh</code> (replace <code>secret</code> with a randomly
	generated client secret):
<pre>
#!/bin/bash
export SECRET=secret
/opt/keycloak/bin/kc.sh bootstrap-admin service --client-id test --client-secret:env=SECRET
exec /opt/keycloak/bin/kc.sh start-dev --log=console --log-console-level=debug --http-enabled true
</pre>
</li>
<li>Dockerfile:
<pre>
FROM keycloak/keycloak
COPY startup.sh /startup.sh
ENTRYPOINT [ "/startup.sh" ]
</pre>
</li>
<li>Environment variables (replace with actual username and password):
<pre>
KC_BOOTSTRAP_ADMIN_USERNAME=admin
KC_BOOTSTRAP_ADMIN_PASSWORD=password
</pre>
</li>
</ol>

<p>Once you have a running KeyCloak instance, log into the admin console and
create a new Realm, User, and Client as documented in their
<a href="https://www.keycloak.org/getting-started/getting-started-docker">
official documentation</a>. When creating the client, ensure that <b>Direct
access grants</b> is checked. You can then set up a mechanism to use to
integrate this authentication method into Slurm.</p>

<p>For example, using <b>JWKS</b>:</p>

<ol>
<li>Create a JSON file with valid RS256 keys:
<pre>
curl "http://10.11.1.23:8080/realms/master/protocol/openid-connect/certs" > /etc/slurm/jwks.json
</pre></li>
<li>Point Slurm to that file for authentication:
<pre>
AuthAltTypes=auth/jwt
AuthAltParameters=jwks=/etc/slurm/jwks.json,userclaimfield=preferred_username
</pre></li>
</ol>

<p>Alternately, using <b>JWT</b>:</p>

<ol>
<li>Create script to generate JWT from KeyCloak:
<pre>
#!/bin/bash
[ -z "$1" -o -z "$2" ] && echo "USAGE:\n$0 {user_name} {user_password}" && exit 1

curl -s \
  -d "client_id=test" \
  -d "client_secret=secret" \
  -d "username=$1" \
  -d "password=$2" \
  -d "grant_type=password" \
  -d "scope=openid" \
  "http://10.11.1.23:8080/realms/master/protocol/openid-connect/token" | \
  jq -r '.id_token'
</pre></li>
<li>You can then use that script to set the JWT variable for Slurm commands or
	REST API requests:
<pre>
env SLURM_JWT=$(get_keycloak_jwt.sh username password) \
sbatch -o none -e none --wrap 'unset SLURM_JWT; srun uptime'
</pre></li>
</ol>

<h3 id="user_mapping">User Mapping
<a class="slurm_link" href="#user_mapping"></a></h3>

<p>Depending on the service used to generate tokens, you may run into issues
mapping the token to a username. Slurm defaults to using the <code>sun</code>
(Slurm UserName) field. If the service uses a different field, you will need to
correct this for it to work with Slurm.</p>

<p><b>Option 1</b>: Change Slurm to use a different field. This can be
customized using <a href="slurm.conf.html#OPT_userclaimfield=">
AuthAltParameters=userclaimfield</a>. For example, using the default field
for <a href="https://www.keycloak.org/">KeyCloak</a>:
<pre>
AuthAltParameters=jwks=/local/path/to/jwks.json,userclaimfield=preferred_username
</pre>
</p>

<p><b>Option 2</b>: Change the identity service to use a different field. In
KeyCloak 25.0, for example, you should find this option under <b>Clients ->
Client details -> Dedicated scopes -> Mapper details</b>. Change the username
mapping to use the <code>sun</code> field.</p>

<h2 id="compatibility">Compatibility
<a class="slurm_link" href="#compatibility"></a>
</h2>
Slurm uses libjwt to view and verify
<a href="https://datatracker.ietf.org/doc/html/rfc7519">RFC7519</a> JWT tokens.
Compliant tokens generated by another solution can be used as long as the
following requirements are met:
<ol>
<li>Required tokens for Slurm are present:
	<ul>
		<li>iat: Unix timestamp of creation date.</li>
		<li>exp: Unix timestamp of expiration date.</li>
		<li>sun or username: Slurm UserName (
			<a href="https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_437">
				POSIX.1-2017 User Name
			</a>).
		</li>
	</ul>
</li>
<li>Tokens are signed with HS256 algorithm compliant to RFC7518. RS256 is also
	supported to verify tokens, although Slurm cannot create them
	directly.</li>
<li>Signing key is provided to slurmctld and slurmdbd to allow decryption of
	the tokens. Slurm currently only supports a single signing key.</li>
</ol>

The following scripts require the installation of the JWT Python module.
This script can serve as an example of what you might do to generate
a jwt key for use with Slurm.
<pre>
#!/usr/bin/env python3
import sys
import os
import pprint
import json
import time
from datetime import datetime, timedelta, timezone

from jwt import JWT
from jwt.jwa import HS256
from jwt.jwk import jwk_from_dict
from jwt.utils import b64decode,b64encode

if len(sys.argv) != 3:
    sys.exit("gen_jwt.py [user name] [expiration time (seconds)]");

with open("/var/spool/slurm/statesave/jwt.key", "rb") as f:
    priv_key = f.read()

signing_key = jwk_from_dict({
    'kty': 'oct',
    'k': b64encode(priv_key)
})

message = {
    "exp": int(time.time() + int(sys.argv[2])),
    "iat": int(time.time()),
    "sun": sys.argv[1]
}

a = JWT()
compact_jws = a.encode(message, signing_key, alg='HS256')
print("SLURM_JWT={}".format(compact_jws))
</pre>

Similarly, the following script can be used as an example of how you might
verify that a jwt key is valid for use with Slurm.
<pre>
#!/usr/bin/env python3
import sys
import os
import pprint
import json
import time
from datetime import datetime, timedelta, timezone

from jwt import JWT
from jwt.jwa import HS256
from jwt.jwk import jwk_from_dict
from jwt.utils import b64decode,b64encode

if len(sys.argv) != 2:
    sys.exit("verify_jwt.py [JWT Token]");

with open("/var/spool/slurm/statesave/jwt.key", "rb") as f:
    priv_key = f.read()

signing_key = jwk_from_dict({
    'kty': 'oct',
    'k': b64encode(priv_key)
})

a = JWT()
b = a.decode(sys.argv[1], signing_key, algorithms=["HS256"])
print(b)
</pre>

<p style="text-align:center;">Last modified 11 June 2025</p>

<!--#include virtual="footer.txt"-->