doc/html/certmgr.shtml - SchedMD/slurm - Git at Google

 <!--#include virtual="header.txt"-->

 <h1><a id="top">TLS Certificate Manager</a></h1>

 <h2 id="Overview">Overview<a class="slurm_link" href="#Overview"></a></h2>

 <p>
 The <code>certmgr</code> plugin interface can be used alongside the
 <code>tls</code> plugin interface to dynamically create and renew signed
 certificates for slurmd/sackd nodes.
 </p>

 <p>
 Signed certificates and accompanying private keys generated with certmgr are
 saved in slurmd's spool directory when they are retrieved from slurmctld, and
 loaded when slurmd starts up.
 </p>

 <h2 id="script">certmgr/script<a class="slurm_link" href="#script"></a></h2>

 <p>
 The <code>certmgr/script</code> plugin allows scripts to be used to perform the
 necessary operations needed to validate node identity and generate signed
 certificates.
 </p>

 <h3 id="script_examples">OpenSSL Example<a class="slurm_link" href="#script_examples"></a></h3>

 <p>
 This is an example using the openssl cli to generate certificate signing
 requests and to sign such requests to create signed certificates. This example
 is not meant to be used in production, and is only mean to show the intended
 responsibilities of each script.
 </p>

 <p>
 In this example, there are a list of things that need to be preloaded on each
 machine before Slurm can do its certificate management. Note that any
 instructions for slurmd also apply to sackd nodes.
 </p>

 <p>
 slurmctld will need access to the CA certificate, and the CA certificate/key
 pair must be owned by <code>SlurmUser</code> (this is NOT recommended in a
 production setting). See the <a href="tls.html#s2n_openssl_example">TLS</a>
 page for more info on how to generate this certificate/key pair.
 </p>

 <p>
 The following scripts need to be created and configured.
 See <a href="slurm.conf.html#OPT_CertmgrParameters">CertmgrParameters</a> for
 more details on each script.
 </p>

 <ul>
 <li><code>get_node_token_script</code></li>
 <li><code>generate_csr_script</code></li>
 <li><code>validate_node_script</code></li>
 <li><code>sign_csr_script</code></li>
 </ul>

 <p>
 slurmctld needs to be able to validate slurmd's certificate signing request.
 This is done via unique tokens that are retrieved on slurmd nodes using
 <code>get_node_token_script</code>, and validated on the slurmctld host using
 <code>validate_node_script</code>.
 </p>

 <p>
 A unique token will to be generated for each slurmd. Each token will be stored
 on its respective slurmd host, as well as in a comprehensive list that contains
 all node tokens on the slurmctld host. This token will be sent from slurmd to
 slurmctld along with the certificate signing request that slurmd will generate
 at runtime, and be validated by slurmctld before slurmctld creates a signed
 certificate. Note that slurmd will not begin to process any RPCs until a signed
 certificate is loaded.
 </p>

 This is a simple example of how these tokens can be generated and stored:
 <pre>
 # generate base64 32 character random token
 base64 /dev/urandom | head -c 32 > ${NODENAME}_token.txt

 # add token to token list
 echo "`cat ${NODENAME}_token.txt`" >> node_token_list.txt
 </pre>

 <p>
 Node <b>n1</b> needs to boot up with <code>n1_token.txt</code> and/or have it
 securely transferred to it. <b>slurmctld</b> needs to have secure access to
 <code>node_token_list.txt</code> in order to validate node tokens with the
 <code>validate_node_script</code>.
 </p>

 <p>
 The <code>get_node_token_script</code>, <code>generate_csr_script</code>, and
 <code>get_node_cert_key_script</code> paths need to point to scripts that exist
 and are executable on slurmd nodes.
 </p>

 <h4 id="get_node_token_script_example">
 get_node_token_script example:<a class="slurm_link" href="#get_node_token_script_example"></a>
 </h4>

 <p>
 Print token to stdout. Return zero exit code for success, and non-zero exit
 code for error.
 </p>

 <pre>
 #!/bin/bash

 # Slurm node name is passed in as arg $1
 TOKEN_PATH=/etc/slurm/certmgr/$1_token.txt

 # Check if token file exists
 if [ ! -f $TOKEN_PATH ]
 then
     echo "$BASH_SOURCE: Failed to resolve token path '$TOKEN_PATH'"
     exit 1
 fi

 # Print token to stdout
 cat $TOKEN_PATH

 # Exit with exit code 0 to indicate success
 exit 0
 </pre>

 <h4 id="generate_csr_script_example">
 generate_csr_script example:<a class="slurm_link" href="#generate_csr_script_example"></a>
 </h4>

 <p>
 Print certificate signing request to stdout. Return zero exit code for success,
 and non-zero exit code for error.
 </p>

 <pre>
 #!/bin/bash

 # Slurm node name is passed in as arg $1
 NODE_PRIVATE_KEY=/etc/slurm/certmgr/$1_private_key.pem

 # Check if node private key file exists
 if [ ! -f $NODE_PRIVATE_KEY ]
 then
     echo "$BASH_SOURCE: Failed to resolve node private key path '$NODE_PRIVATE_KEY'"
     exit 1
 fi

 # Generate CSR using node private key and print CSR to stdout
 openssl req -new -key $NODE_PRIVATE_KEY \
     -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=$1"

 # Check exit code from openssl
 if [ $? -ne 0 ]
 then
     echo "$BASH_SOURCE: Failed to generate CSR"
     exit 1
 fi

 # Exit with exit code 0 to indicate success
 exit 0
 </pre>

 <h4 id="get_node_cert_key_script">
 get_node_cert_key_script example:<a class="slurm_link" href="#get_node_cert_key_script"></a>
 </h4>

 <p>
 Print private key used to generate CSR to stdout. Return zero exit code for
 success, and non-zero exit code for error.
 </p>

 <pre>
 #!/bin/bash

 # Slurm node name is passed in as arg $1
 NODE_PRIVATE_KEY=/etc/slurm/certmgr/$1_cert_key.pem

 # Check if node private key file exists
 if [ ! -f $NODE_PRIVATE_KEY ]
 then
     echo "$BASH_SOURCE: Failed to resolve node private key path '$NODE_PRIVATE_KEY'"
     exit 1
 fi

 cat $NODE_PRIVATE_KEY

 # Exit with exit code 0 to indicate success
 exit 0
 </pre>

 <p>
 The <code>validate_node_script</code> and <code>sign_csr_script</code> paths
 need to point to scripts that exist and are executable on <b>slurmctld</b>.
 </p>

 <h4 id="validate_node_script_example">
 validate_node_script example:<a class="slurm_link" href="#validate_node_script_example"></a>
 </h4>

 <p>
 Return zero exit code for valid node tokens, and non-zero exit code  for
 invalid node tokens or other errors.
 </p>

 <pre>
 #!/bin/bash

 # Node's unique token is passed in as arg $1
 NODE_TOKEN=$1
 NODE_TOKEN_LIST_FILE=/etc/slurm/certmgr/node_token_list.txt

 # Check if node token list file exists
 if [ ! -f $NODE_TOKEN_LIST ]
 then
     echo "$BASH_SOURCE: Failed to resolve node token list path '$NODE_TOKEN_LIST'"
     exit 1
 fi

 # Check if unique node token is in token list file
 grep $1 $NODE_TOKEN_LIST_FILE

 # Check exit code from grep to see if token was found
 if [ $? -ne 0 ]
 then
     echo "$BASH_SOURCE: Failed to validate token '$NODE_TOKEN'"
     exit 1
 fi

 # Exit with exit code 0 to indicate success (node token is valid)
 exit 0
 </pre>

 <h4 id="sign_csr_script_example">
 sign_csr_script example:<a class="slurm_link" href="#sign_csr_script_example"></a>
 </h4>

 <p>
 Print signed certificate to stdout. Return zero exit code for success, and
 non-zero exit code for error.
 </p>

 <pre>
 #!/bin/bash

 # Certificate signing request is passed in as arg $1
 CSR=$1
 CA_CERT=/etc/slurm/certmgr/root_cert.pem
 CA_KEY=/etc/slurm/certmgr/root_key.pem

 # Check if CA certificate file exists
 if [ ! -f $CA_CERT ]
 then
     echo "$BASH_SOURCE: Failed to resolve CA certificate path '$CA_CERT'"
     exit 1
 fi

 # Check CA private key permissions
 if [ `stat -c "%a" $CA_KEY` -ne $KEY_PERMISSIONS ]
 then
     echo "$BASH_SOURCE: Bad permissions for CA private key at '$CA_KEY'. Permissions should be $KEY_PERMISSIONS"
     exit 1
 fi

 # Sign CSR using CA certificate and CA private key and print signed cert to stdout
 openssl x509 -req -CA $CA_CERT -CAkey $CA_KEY 2>/dev/null <<< $CSR

 # Check exit code from openssl
 if [ $? -ne 0 ]
 then
     echo "$BASH_SOURCE: Failed to generate signed certificate"
     exit 1
 fi

 # Exit with exit code 0 to indicate success
 exit 0
 </pre>

 <p>
 If everything is configured correctly, the following lines should appear in the
 slurmd and slurmctld logs with the
 <a href="slurm.conf.html#OPT_TLS">DebugFlags=TLS</a> setting.
 </p>

 <p>slurmd:</p>
 <pre>
 slurmd: certmgr/script: certmgr_p_get_node_token: TLS: Successfully retrieved unique node token
 slurmd: certmgr/script: certmgr_p_generate_csr: TLS: Successfully generated csr:
 -----BEGIN CERTIFICATE REQUEST-----
 . . .
 -----END CERTIFICATE REQUEST-----
 </pre>

 <p>slurmctld:</p>
 <pre>
 slurmctld: certmgr/script: certmgr_p_sign_csr: TLS: Successfully validated node token
 slurmctld: certmgr/script: certmgr_p_sign_csr: TLS: Successfully generated signed certificate:
 -----BEGIN CERTIFICATE-----
 . . .
 -----END CERTIFICATE-----
 </pre>

 <p>slurmd:</p>
 <pre>
 slurmd: TLS: Successfully got signed certificate from slurmctld:
 -----BEGIN CERTIFICATE-----
 . . .
 -----END CERTIFICATE-----
 </pre>

 <p>
 <a href="slurm.conf.html#OPT_AuditTLS">DebugFlags=AuditTLS</a> can also be used
 to show less verbose logs of certificate renewal.
 </p>

 <p style="text-align:center;">Last modified 06 July 2025</p>

 <!--#include virtual="footer.txt"-->
	<!--#include virtual="header.txt"-->

	<h1><a id="top">TLS Certificate Manager</a></h1>

	<h2 id="Overview">Overview<a class="slurm_link" href="#Overview"></a></h2>

	<p>
	The <code>certmgr</code> plugin interface can be used alongside the
	<code>tls</code> plugin interface to dynamically create and renew signed
	certificates for slurmd/sackd nodes.
	</p>

	<p>
	Signed certificates and accompanying private keys generated with certmgr are
	saved in slurmd's spool directory when they are retrieved from slurmctld, and
	loaded when slurmd starts up.
	</p>

	<h2 id="script">certmgr/script<a class="slurm_link" href="#script"></a></h2>

	<p>
	The <code>certmgr/script</code> plugin allows scripts to be used to perform the
	necessary operations needed to validate node identity and generate signed
	certificates.
	</p>

	<h3 id="script_examples">OpenSSL Example<a class="slurm_link" href="#script_examples"></a></h3>

	<p>
	This is an example using the openssl cli to generate certificate signing
	requests and to sign such requests to create signed certificates. This example
	is not meant to be used in production, and is only mean to show the intended
	responsibilities of each script.
	</p>

	<p>
	In this example, there are a list of things that need to be preloaded on each
	machine before Slurm can do its certificate management. Note that any
	instructions for slurmd also apply to sackd nodes.
	</p>

	<p>
	slurmctld will need access to the CA certificate, and the CA certificate/key
	pair must be owned by <code>SlurmUser</code> (this is NOT recommended in a
	production setting). See the <a href="tls.html#s2n_openssl_example">TLS</a>
	page for more info on how to generate this certificate/key pair.
	</p>

	<p>
	The following scripts need to be created and configured.
	See <a href="slurm.conf.html#OPT_CertmgrParameters">CertmgrParameters</a> for
	more details on each script.
	</p>

	<ul>
	<li><code>get_node_token_script</code></li>
	<li><code>generate_csr_script</code></li>
	<li><code>validate_node_script</code></li>
	<li><code>sign_csr_script</code></li>
	</ul>

	<p>
	slurmctld needs to be able to validate slurmd's certificate signing request.
	This is done via unique tokens that are retrieved on slurmd nodes using
	<code>get_node_token_script</code>, and validated on the slurmctld host using
	<code>validate_node_script</code>.
	</p>

	<p>
	A unique token will to be generated for each slurmd. Each token will be stored
	on its respective slurmd host, as well as in a comprehensive list that contains
	all node tokens on the slurmctld host. This token will be sent from slurmd to
	slurmctld along with the certificate signing request that slurmd will generate
	at runtime, and be validated by slurmctld before slurmctld creates a signed
	certificate. Note that slurmd will not begin to process any RPCs until a signed
	certificate is loaded.
	</p>

	This is a simple example of how these tokens can be generated and stored:
	<pre>
	# generate base64 32 character random token
	base64 /dev/urandom \| head -c 32 > ${NODENAME}_token.txt

	# add token to token list
	echo "`cat ${NODENAME}_token.txt`" >> node_token_list.txt
	</pre>

	<p>
	Node <b>n1</b> needs to boot up with <code>n1_token.txt</code> and/or have it
	securely transferred to it. <b>slurmctld</b> needs to have secure access to
	<code>node_token_list.txt</code> in order to validate node tokens with the
	<code>validate_node_script</code>.
	</p>

	<p>
	The <code>get_node_token_script</code>, <code>generate_csr_script</code>, and
	<code>get_node_cert_key_script</code> paths need to point to scripts that exist
	and are executable on slurmd nodes.
	</p>

	<h4 id="get_node_token_script_example">
	get_node_token_script example:<a class="slurm_link" href="#get_node_token_script_example"></a>
	</h4>

	<p>
	Print token to stdout. Return zero exit code for success, and non-zero exit
	code for error.
	</p>

	<pre>
	#!/bin/bash

	# Slurm node name is passed in as arg $1
	TOKEN_PATH=/etc/slurm/certmgr/$1_token.txt

	# Check if token file exists
	if [ ! -f $TOKEN_PATH ]
	then
	echo "$BASH_SOURCE: Failed to resolve token path '$TOKEN_PATH'"
	exit 1
	fi

	# Print token to stdout
	cat $TOKEN_PATH

	# Exit with exit code 0 to indicate success
	exit 0
	</pre>

	<h4 id="generate_csr_script_example">
	generate_csr_script example:<a class="slurm_link" href="#generate_csr_script_example"></a>
	</h4>

	<p>
	Print certificate signing request to stdout. Return zero exit code for success,
	and non-zero exit code for error.
	</p>

	<pre>
	#!/bin/bash

	# Slurm node name is passed in as arg $1
	NODE_PRIVATE_KEY=/etc/slurm/certmgr/$1_private_key.pem

	# Check if node private key file exists
	if [ ! -f $NODE_PRIVATE_KEY ]
	then
	echo "$BASH_SOURCE: Failed to resolve node private key path '$NODE_PRIVATE_KEY'"
	exit 1
	fi

	# Generate CSR using node private key and print CSR to stdout
	openssl req -new -key $NODE_PRIVATE_KEY \
	-subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=$1"

	# Check exit code from openssl
	if [ $? -ne 0 ]
	then
	echo "$BASH_SOURCE: Failed to generate CSR"
	exit 1
	fi

	# Exit with exit code 0 to indicate success
	exit 0
	</pre>

	<h4 id="get_node_cert_key_script">
	get_node_cert_key_script example:<a class="slurm_link" href="#get_node_cert_key_script"></a>
	</h4>

	<p>
	Print private key used to generate CSR to stdout. Return zero exit code for
	success, and non-zero exit code for error.
	</p>

	<pre>
	#!/bin/bash

	# Slurm node name is passed in as arg $1
	NODE_PRIVATE_KEY=/etc/slurm/certmgr/$1_cert_key.pem

	# Check if node private key file exists
	if [ ! -f $NODE_PRIVATE_KEY ]
	then
	echo "$BASH_SOURCE: Failed to resolve node private key path '$NODE_PRIVATE_KEY'"
	exit 1
	fi

	cat $NODE_PRIVATE_KEY

	# Exit with exit code 0 to indicate success
	exit 0
	</pre>

	<p>
	The <code>validate_node_script</code> and <code>sign_csr_script</code> paths
	need to point to scripts that exist and are executable on <b>slurmctld</b>.
	</p>

	<h4 id="validate_node_script_example">
	validate_node_script example:<a class="slurm_link" href="#validate_node_script_example"></a>
	</h4>

	<p>
	Return zero exit code for valid node tokens, and non-zero exit code for
	invalid node tokens or other errors.
	</p>

	<pre>
	#!/bin/bash

	# Node's unique token is passed in as arg $1
	NODE_TOKEN=$1
	NODE_TOKEN_LIST_FILE=/etc/slurm/certmgr/node_token_list.txt

	# Check if node token list file exists
	if [ ! -f $NODE_TOKEN_LIST ]
	then
	echo "$BASH_SOURCE: Failed to resolve node token list path '$NODE_TOKEN_LIST'"
	exit 1
	fi

	# Check if unique node token is in token list file
	grep $1 $NODE_TOKEN_LIST_FILE

	# Check exit code from grep to see if token was found
	if [ $? -ne 0 ]
	then
	echo "$BASH_SOURCE: Failed to validate token '$NODE_TOKEN'"
	exit 1
	fi

	# Exit with exit code 0 to indicate success (node token is valid)
	exit 0
	</pre>

	<h4 id="sign_csr_script_example">
	sign_csr_script example:<a class="slurm_link" href="#sign_csr_script_example"></a>
	</h4>

	<p>
	Print signed certificate to stdout. Return zero exit code for success, and
	non-zero exit code for error.
	</p>

	<pre>
	#!/bin/bash

	# Certificate signing request is passed in as arg $1
	CSR=$1
	CA_CERT=/etc/slurm/certmgr/root_cert.pem
	CA_KEY=/etc/slurm/certmgr/root_key.pem

	# Check if CA certificate file exists
	if [ ! -f $CA_CERT ]
	then
	echo "$BASH_SOURCE: Failed to resolve CA certificate path '$CA_CERT'"
	exit 1
	fi

	# Check CA private key permissions
	if [ `stat -c "%a" $CA_KEY` -ne $KEY_PERMISSIONS ]
	then
	echo "$BASH_SOURCE: Bad permissions for CA private key at '$CA_KEY'. Permissions should be $KEY_PERMISSIONS"
	exit 1
	fi

	# Sign CSR using CA certificate and CA private key and print signed cert to stdout
	openssl x509 -req -CA $CA_CERT -CAkey $CA_KEY 2>/dev/null <<< $CSR

	# Check exit code from openssl
	if [ $? -ne 0 ]
	then
	echo "$BASH_SOURCE: Failed to generate signed certificate"
	exit 1
	fi

	# Exit with exit code 0 to indicate success
	exit 0
	</pre>

	<p>
	If everything is configured correctly, the following lines should appear in the
	slurmd and slurmctld logs with the
	<a href="slurm.conf.html#OPT_TLS">DebugFlags=TLS</a> setting.
	</p>

	<p>slurmd:</p>
	<pre>
	slurmd: certmgr/script: certmgr_p_get_node_token: TLS: Successfully retrieved unique node token
	slurmd: certmgr/script: certmgr_p_generate_csr: TLS: Successfully generated csr:
	-----BEGIN CERTIFICATE REQUEST-----
	. . .
	-----END CERTIFICATE REQUEST-----
	</pre>

	<p>slurmctld:</p>
	<pre>
	slurmctld: certmgr/script: certmgr_p_sign_csr: TLS: Successfully validated node token
	slurmctld: certmgr/script: certmgr_p_sign_csr: TLS: Successfully generated signed certificate:
	-----BEGIN CERTIFICATE-----
	. . .
	-----END CERTIFICATE-----
	</pre>

	<p>slurmd:</p>
	<pre>
	slurmd: TLS: Successfully got signed certificate from slurmctld:
	-----BEGIN CERTIFICATE-----
	. . .
	-----END CERTIFICATE-----
	</pre>

	<p>
	<a href="slurm.conf.html#OPT_AuditTLS">DebugFlags=AuditTLS</a> can also be used
	to show less verbose logs of certificate renewal.
	</p>

	<p style="text-align:center;">Last modified 06 July 2025</p>

	<!--#include virtual="footer.txt"-->