doc/html/rest_quickstart.shtml - SchedMD/slurm - Git at Google


            
            <!--#include virtual="header.txt"-->


            <h1>REST API Quick Start Guide</h1>


            <p>Slurm provides a <a href="https://restfulapi.net/">REST API</a> through the

            
            slurmrestd daemon, using <a href="jwt.html">JSON Web Tokens</a> for

            
            authentication. This daemon is designed to allow clients to communicate with

            
            Slurm via a REST API (in addition to the command line interface (CLI) or C API).

            
            This page provides a brief tutorial for setting up these components.</p>


            <p>

            
            See also:

            
            <ul>

            
            <li><a href="rest.html">REST API Details</a></li>

            
            <li><a href="rest_api.html">REST API Methods and Models</a></li>

            
            <li><a href="slurmrestd.html">slurmrestd man page</a></li>

            
            <li><a href="openapi_release_notes.html">OpenAPI Plugin Release Notes</a></li>

            
            <li><a href="rest_clients.html">REST API Client Guide</a></li>

            
            </ul>

            
            </p>


            <h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>

            
            <ul>

            
            <li><a href="#prereq">Prerequisites</a></li>

            
            <li><a href="#quick_start">Quick Start</a>

            
              <ul>

            
              <li><a href="#systemd">Running with systemd</a></li>

            
              <li><a href="#customization">Customizing slurmrestd.service</a></li>

            
              </ul>

            
            </li>

            
            <li><a href="#basic_usage">Basic Usage</a>

            
              <ul>

            
              <li><a href="#tokens">Token management</a></li>

            
              </ul>

            
            </li>

            
            <li><a href="#advanced_usage">Advanced Usage</a></li>

            
            <li><a href="#common_issues">Common Issues</a>

            
              <ul>

            
              <li><a href="#bind_socket">Unable to bind socket</a></li>

            
              <li><a href="#connection_refused">Connection refused</a></li>

            
              <li><a href="#authentication_error">Protocol authentication error (HTTP 500)

            
                  </a></li>

            
              <li><a href="#invalid_url">Unable to find requested URL (HTTP 404)</a></li>

            
              <li><a href="#rejected_token">Rejecting thread config token (HTTP 401)

            
                  </a></li>

            
              <li><a href="#unexpected_character">Unexpected URL character (HTTP 400)

            
                  </a></li>

            
              <li><a href="#slurm_commands">Other Slurm commands not working</a></li>

            
              </ul>

            
            </li>

            
            </ul>


            <h2 id="prereq">Prerequisites

            
            <a class="slurm_link" href="#prereq"></a></h2>

            
            <p>The following development libraries are required at compile time

            
            in order for slurmrestd to be compiled (minimum versions are on the related

            
            software page linked below):</p>

            
            <ul>

            
            	<li><a href="related_software.html#httpparser">HTTP Parser</a></li>

            
            	<li><a href="related_software.html#json">JSON-C</a></li>

            
            </ul>


            <p>The following development libraries are optional; if present at compile time

            
            the related functionality will be available:</p>


            <ul>

            
            	<li><a href="related_software.html#yaml">YAML</a> (YAML support)</li>

            
            	<li><a href="related_software.html#jwt">JWT</a>

            
                (authentication for socket-based clients)</li>

            
            	<li><a href="tls.html#s2n">s2n</a> (TLS communications)</li>

            
            </ul>


            <p>It is generally recommended that you have

            
            <a href="slurmdbd.html">slurmdbd</a> set up for

            
            <a href="accounting.html">accounting</a>. Without slurmdbd you may need to

            
            start slurmrestd with the <a href="slurmrestd.html#OPT_-s-<plugin>">-s</a>

            
            flag to tell it not to load the slurmdbd plugin.</p>


            <h2 id="quick_start">Quick Start

            
            <a class="slurm_link" href="#quick_start"></a></h2>


            <p>This may be done on a dedicated REST API machine or

            
            your existing 'slurmctld' machine, depending on demand.

            
            If you have multiple clusters, you will need a unique instance of

            
            <b>slurmrestd</b> for each cluster.</p>


            <ol>

            
            <li>Install components for slurmrestd

            
            <ul>

            
            <li>DEB: <code>slurm-smd slurm-smd-slurmrestd</code></li>

            
            <li>RPM: <code>slurm slurm-slurmrestd</code> (requires

            
            <code>--with slurmrestd</code> at build time)</li>

            
            </ul></li>

            
            <li>Set up <a href="jwt.html">JSON Web Tokens</a> for authentication</li>

            
            <li>Ensure <code>/etc/slurm/slurm.conf</code> is present and correct for your

            
            cluster (see <a href="quickstart_admin.html">Quick Start Admin Guide</a> and

            
            <a href="slurm.conf.html">slurm.conf man page</a>)</li>

            
            <li>Run <b>slurmrestd</b> (see <a href="#systemd">below</a> for systemd

            
            instructions) on your preferred [HOST]:PORT combination

            
            (':6820' is the default for production)

            
            <pre>export SLURM_JWT=daemon

            
            export SLURMRESTD_DEBUG=debug

            
            slurmrestd &lt;host&gt;:&lt;port&gt;

            
            </pre>

            
            Adjust SLURMRESTD_DEBUG to the desired level of output (as described on the

            
            <a href="slurmrestd.html#OPT_SLURMRESTD_DEBUG">man page</a>)</li>

            
            </ol>


            <h3 id="systemd">Running with systemd

            
            <a class="slurm_link" href="#systemd"></a></h3>


            <p>Slurm ships with a <b>slurmrestd</b> service unit for systemd,

            
            however, it might require some additional setup to run properly.

            
            This section assumes you have either installed slurmrestd using DEB/RPM packages

            
            or built it manually such that the files are in the same places.

            
            <br><b>Note</b> the versions associated with certain steps in the instructions

            
            below; these steps should be ignored on other versions.</p>

            
            <ol>

            
              <li>Create a local service account and group of <code>slurmrestd</code> to

            
              run the slurmrestd daemon. To prevent privilege escalation, the user account

            
              should be:

            
              <ul>

            
                <li>Not root or SlurmUser</li>

            
                <li>Not used by real users or for any other functions</li>

            
                <li>Not granted any special permissions</li>

            
              </ul>

            
              <pre>sudo useradd -M -r -s /usr/sbin/nologin -U slurmrestd</pre></li>

            
              <li>If needed, sites can run <code>systemctl edit slurmrestd</code> to

            
              override the default User and Group to the site specific user and group:

            
              <pre>[Service]

            
              User=slurmrestd

            
              Group=slurmrestd</pre></li>

            
              </ul>

            
            </li>

            
            <li>(Slurm <b>24.05</b> and newer) Optional: Customize the socket for

            
            slurmrestd. By default it will listen only on TCP port 6820. You may change

            
            this behavior by changing <code>SLURMRESTD_LISTEN</code>

            
            (see <a href="#customization">Customizing slurmrestd.service</a>).</li>

            
            <li>(Slurm <b>23.11</b> and earlier) Configure the socket for slurmrestd. This

            
            may be accomplished by creating/changing permissions on the parent directory

            
            and/or changing the path to the socket in the service file.

            
            <ul>

            
            <li><b>Permissions</b>: The user running the service must have write+execute

            
            permissions on the directory that will contain the UNIX socket</li>

            
            <li><b>Changing socket</b>: On Slurm 23.11, the way to change or disable the

            
            socket is to modify the 'ExecStart' line of the service

            
            <ol>

            
            <li>Run <code>systemctl edit slurmrestd</code></li>

            
            <li>Add the following contents to the <code>[Service]</code> section:

            
            <pre>

            
            ExecStart=

            
            ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS

            
            Environment=SLURMRESTD_LISTEN=:6820

            
            </pre></li>

            
            <li>Adjust the assignment of SLURMRESTD_LISTEN to contain the socket(s) you want

            
            the daemon to listen on.</li>

            
            <li>After a future upgrade to Slurm 24.05+, the 'ExecStart' overrides will be

            
            unnecessary but will not conflict with the newer version.</li>

            
            </ol>

            
            </li>

            
            </ul>

            
            </li>

            
            </ol>


            <h3 id="customization">Customizing slurmrestd.service

            
            <a class="slurm_link" href="#customization"></a></h3>


            <p>The Slurm 24.05 release changes the operation of

            
            the default service file and may break existing overrides. If you have

            
            overridden <code>ExecStart=</code> to contain any TCP/UNIX sockets directly, it

            
            will cause the service to fail if it duplicates any sockets contained in

            
            SLURMRESTD_LISTEN. These overrides will need to be changed after upgrading.</p>


            <p>The default <code>slurmrestd.service</code> file has two intended ways of

            
            customizing its operation:</p>


            <ol>

            
            <li><b>Environment files</b>:

            
            The service will read environment variables from two files:

            
            <code>/etc/default/slurmrestd</code> and <code>/etc/sysconfig/slurmrestd</code>.

            
            You may set any environment variables recognized by

            
            <a href="slurmrestd.html#SECTION_ENVIRONMENT-VARIABLES">slurmrestd</a>,

            
            but the following are particularly relevant:

            
            <ul>

            
            <li><b>SLURMRESTD_OPTIONS</b>: CLI options to add to the slurmrestd command

            
            (see <a href="slurmrestd.html">slurmrestd</a>)</li>

            
            <li><b>SLURMRESTD_LISTEN</b>: Comma-delimited list of host:port pairs or

            
            unix:$SOCKET_PATH sockets to listen on

            
            <br><b>NOTE</b>: If this duplicates what is already set in the

            
            'ExecStart' line in the service file, it will fail. Starting in Slurm 24.05,

            
            the default service file contains no sockets in 'ExecStart' and fully relies on

            
            this variable to contain the desired sockets.</li>

            
            </ul>

            
            </li>

            
            <li><b>Service editing</b>: Systemd has a built in way to edit services

            
            by running <code>systemctl edit slurmrestd</code>.

            
            <ul>

            
            <li>This will create an override file in '/etc/systemd/' containing directives

            
            that will add to or replace directives in the default unit in '/lib/systemd/'.

            
            </li>

            
            <li>The override file must have the appropriate section declaration(s)

            
            for the directives you use (e.g., <code>[Service]</code>).</li>

            
            <li>Environment variables may be set with <code>Environment=NAME=value</code>

            
            (refer to systemd documentation for more details)</li>

            
            <li>Changes may be reverted by running <code>systemctl revert slurmrestd</code>

            
            </li>

            
            </ul>

            
            </li>

            
            </ol>


            <h2 id="basic_usage">Basic Usage

            
            <a class="slurm_link" href="#basic_usage"></a></h2>


            <ol>

            
            <li>Find the latest supported API version

            
            <pre>slurmrestd -d list</pre></li>

            
            <li>Get an authentication token for JWT

            
            <pre>unset SLURM_JWT; export $(scontrol token)</pre>

            
            <ul>

            
            <li>This ensures an old token doesn't prevent a new one from being issued</li>

            
            <li>By default, tokens will expire after 1800 seconds (30 minutes).

            
            Add <code>lifespan=SECONDS</code> to the 'scontrol' command to change this.</li>

            
            </ul>

            
            </li>

            
            <li>Run a basic curl command to hit the API when listening on a TCP host:port

            
            <pre>curl -s -o "/tmp/curl.log" -k -vvvv \

            
            -H X-SLURM-USER-TOKEN:$SLURM_JWT \

            
            -X GET 'http://&lt;server&gt;:&lt;port&gt;/slurm/v0.0.&lt;api-version&gt;/diag'

            
            </pre>

            
            <ul>

            
            <li>Replace the <b>server</b>, <b>port</b>, and <b>api-version</b>

            
            with the appropriate values.</li>

            
            <li>Examine the output to ensure the response was <b>200 OK</b>,

            
            and examine <b>/tmp/curl.log</b> for a valid JSON response.</li>

            
            <li>Try other endpoints described in the <a href="rest_api.html">API Methods

            
            and Models</a>. Change <b>GET</b> to the correct method for the endpoint.</li>

            
            </ul>

            
            </li>

            
            <li>Alternate command to use the UNIX socket instead

            
            <pre>

            
            curl -s -o "/tmp/curl.log" -k -vvvv \

            
            -H X-SLURM-USER-TOKEN:$SLURM_JWT \

            
            --unix-socket /path/to/slurmrestd.socket \

            
            'http://&lt;server&gt;/slurm/v0.0.&lt;api-version&gt;/diag'

            
            </pre>

            
            <ul>

            
            <li>Replace the <b>path</b>, <b>server</b>, and <b>api-version</b>

            
            with the appropriate values.</li>

            
            <li>Examine the output to ensure the response was <b>200 OK</b>,

            
            and examine <b>/tmp/curl.log</b> for a valid JSON response.</li>

            
            </ul>

            
            </li>

            
            </ol>


            <h3 id="tokens">Token management

            
            <a class="slurm_link" href="#tokens"></a></h3>


            <p>This guide provides a simple overview using <code>scontrol</code> to

            
            obtain tokens. This is a basic introductory approach that in many cases

            
            should be disabled in favor of more sophisticated token management.

            
            Refer to the <a href="jwt.html">JWT page</a> for more details.</p>


            <h2 id="advanced_usage">Advanced Usage

            
            <a class="slurm_link" href="#advanced_usage"></a></h2>


            <p>Information about ways to further customize and configure slurmrestd,

            
            including authentication methods, run modes, plugins, high availability, proxies,

            
            and Python client is found on the <a href="rest.html">REST API Details</a> page.</p>


            <h2 id="common_issues">Common Issues

            
            <a class="slurm_link" href="#common_issues"></a></h2>


            <p>In general, look out for these things:</p>


            <ol>

            
            <li>Validity of authentication token in <code>SLURM_JWT</code></li>

            
            <li>Hostname and port number</li>

            
            <li>API version and endpoint</li>

            
            <li>Log output of slurmrestd</li>

            
            </ol>


            <h3 id="bind_socket">Unable to bind socket

            
            <a class="slurm_link" href="#bind_socket"></a></h3>


            <p>This may be due to a permissions issue while attempting to set up the socket.

            
            Check the log output from slurmrestd for the path to the socket.

            
            Ensure that the user running the slurmrestd service has permissions to the

            
            parent directory of the configured socket path, or change/remove the socket path

            
            as <a href="#systemd">described above</a>.</p>


            <p>If it says "<b>Address already in use</b>", check the command being run

            
            and the contents of "SLURMRESTD_LISTEN" for duplicates of the same TCP or UNIX

            
            socket.</p>


            <h3 id="connection_refused">Connection refused

            
            <a class="slurm_link" href="#connection_refused"></a></h3>


            <p>Verify that slurmrestd is running and listening on the port you are

            
            attempting to connect to.</p>


            <h3 id="authentication_error">Protocol authentication error (HTTP 500)

            
            <a class="slurm_link" href="#authentication_error"></a></h3>


            <p>One common authentication problem is an expired token. Request a new one:

            
            <pre>unset SLURM_JWT; export $(scontrol token)</pre></p>


            <p>This solution also applies to an HTTP 401 error caused by no authentication

            
            token being sent at all. This may appear in the slurmrestd logs as

            
            "Authentication does not apply to request."</p>


            <p>Otherwise, consult the logs on the <b>slurmctld</b> and <b>slurmdbd</b>.</p>


            <h3 id="invalid_url">Unable to find requested URL (HTTP 404)

            
            <a class="slurm_link" href="#invalid_url"></a></h3>


            <p>Check the <a href="rest_api.html">API Methods and Models</a> page to ensure

            
            you're using a valid URL and the correct method for it. Pay attention to the

            
            path as there are different endpoints for <b>slurm</b> and <b>slurmdbd</b>.</p>


            <h3 id="rejected_token">Rejecting thread config token (HTTP 401)

            
            <a class="slurm_link" href="#rejected_token"></a></h3>


            <p>Check that slurmrestd has loaded the <b>auth/jwt</b> plugin.

            
            You should see a debug message like this:

            
            <pre>slurmrestd: debug:  auth/jwt: init: JWT authentication plugin loaded</pre>

            
            If it didn't load jwt, run this in the terminal you're using for slurmrestd:

            
            <pre>export SLURM_JWT=daemon</pre>

            
            </p>


            <h3 id="unexpected_character">Unexpected URL character (HTTP 400)

            
            <a class="slurm_link" href="#unexpected_character"></a></h3>


            <p>Check the request URL and slurmrestd logs for characters that may be causing

            
            the URL to be parsed incorrectly. Use the appropriate URL encoding sequence in

            
            place of the problematic character (e.g., <b>/</b> = <b>%2F</b>).

            
            <pre>

            
            ... -X GET "localhost:8080/slurmdb/v0.0.40/jobs?submit_time=02/28/24"

            
            ### 400 BAD REQUEST

            
            ... -X GET "localhost:8080/slurmdb/v0.0.40/jobs?submit_time=02%2F28%2F24"

            
            ### 200 OK

            
            </pre>

            
            </p>


            <h3 id="slurm_commands">Other slurm commands not working

            
            <a class="slurm_link" href="#slurm_commands"></a></h3>


            <p>If SLURM_JWT is set, other slurm commands will attempt to use JWT

            
            authentication, causing failures. This can be fixed by clearing the variable:

            
            <pre>unset SLURM_JWT</pre></p>


            <!--#include virtual="footer.txt"-->
	<!--#include virtual="header.txt"-->

	<h1>REST API Quick Start Guide</h1>

	<p>Slurm provides a <a href="https://restfulapi.net/">REST API</a> through the
	slurmrestd daemon, using <a href="jwt.html">JSON Web Tokens</a> for
	authentication. This daemon is designed to allow clients to communicate with
	Slurm via a REST API (in addition to the command line interface (CLI) or C API).
	This page provides a brief tutorial for setting up these components.</p>

	<p>
	See also:
	<ul>
	<li><a href="rest.html">REST API Details</a></li>
	<li><a href="rest_api.html">REST API Methods and Models</a></li>
	<li><a href="slurmrestd.html">slurmrestd man page</a></li>
	<li><a href="openapi_release_notes.html">OpenAPI Plugin Release Notes</a></li>
	<li><a href="rest_clients.html">REST API Client Guide</a></li>
	</ul>
	</p>

	<h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2>
	<ul>
	<li><a href="#prereq">Prerequisites</a></li>
	<li><a href="#quick_start">Quick Start</a>
	<ul>
	<li><a href="#systemd">Running with systemd</a></li>
	<li><a href="#customization">Customizing slurmrestd.service</a></li>
	</ul>
	</li>
	<li><a href="#basic_usage">Basic Usage</a>
	<ul>
	<li><a href="#tokens">Token management</a></li>
	</ul>
	</li>
	<li><a href="#advanced_usage">Advanced Usage</a></li>
	<li><a href="#common_issues">Common Issues</a>
	<ul>
	<li><a href="#bind_socket">Unable to bind socket</a></li>
	<li><a href="#connection_refused">Connection refused</a></li>
	<li><a href="#authentication_error">Protocol authentication error (HTTP 500)
	</a></li>
	<li><a href="#invalid_url">Unable to find requested URL (HTTP 404)</a></li>
	<li><a href="#rejected_token">Rejecting thread config token (HTTP 401)
	</a></li>
	<li><a href="#unexpected_character">Unexpected URL character (HTTP 400)
	</a></li>
	<li><a href="#slurm_commands">Other Slurm commands not working</a></li>
	</ul>
	</li>
	</ul>

	<h2 id="prereq">Prerequisites
	<a class="slurm_link" href="#prereq"></a></h2>
	<p>The following development libraries are required at compile time
	in order for slurmrestd to be compiled (minimum versions are on the related
	software page linked below):</p>
	<ul>
	<li><a href="related_software.html#httpparser">HTTP Parser</a></li>
	<li><a href="related_software.html#json">JSON-C</a></li>
	</ul>

	<p>The following development libraries are optional; if present at compile time
	the related functionality will be available:</p>

	<ul>
	<li><a href="related_software.html#yaml">YAML</a> (YAML support)</li>
	<li><a href="related_software.html#jwt">JWT</a>
	(authentication for socket-based clients)</li>
	<li><a href="tls.html#s2n">s2n</a> (TLS communications)</li>
	</ul>

	<p>It is generally recommended that you have
	<a href="slurmdbd.html">slurmdbd</a> set up for
	<a href="accounting.html">accounting</a>. Without slurmdbd you may need to
	start slurmrestd with the <a href="slurmrestd.html#OPT_-s-<plugin>">-s</a>
	flag to tell it not to load the slurmdbd plugin.</p>

	<h2 id="quick_start">Quick Start
	<a class="slurm_link" href="#quick_start"></a></h2>

	<p>This may be done on a dedicated REST API machine or
	your existing 'slurmctld' machine, depending on demand.
	If you have multiple clusters, you will need a unique instance of
	<b>slurmrestd</b> for each cluster.</p>

	<ol>
	<li>Install components for slurmrestd
	<ul>
	<li>DEB: <code>slurm-smd slurm-smd-slurmrestd</code></li>
	<li>RPM: <code>slurm slurm-slurmrestd</code> (requires
	<code>--with slurmrestd</code> at build time)</li>
	</ul></li>
	<li>Set up <a href="jwt.html">JSON Web Tokens</a> for authentication</li>
	<li>Ensure <code>/etc/slurm/slurm.conf</code> is present and correct for your
	cluster (see <a href="quickstart_admin.html">Quick Start Admin Guide</a> and
	<a href="slurm.conf.html">slurm.conf man page</a>)</li>
	<li>Run <b>slurmrestd</b> (see <a href="#systemd">below</a> for systemd
	instructions) on your preferred [HOST]:PORT combination
	(':6820' is the default for production)
	<pre>export SLURM_JWT=daemon
	export SLURMRESTD_DEBUG=debug
	slurmrestd <host>:<port>
	</pre>
	Adjust SLURMRESTD_DEBUG to the desired level of output (as described on the
	<a href="slurmrestd.html#OPT_SLURMRESTD_DEBUG">man page</a>)</li>
	</ol>

	<h3 id="systemd">Running with systemd
	<a class="slurm_link" href="#systemd"></a></h3>

	<p>Slurm ships with a <b>slurmrestd</b> service unit for systemd,
	however, it might require some additional setup to run properly.
	This section assumes you have either installed slurmrestd using DEB/RPM packages
	or built it manually such that the files are in the same places.
	<br><b>Note</b> the versions associated with certain steps in the instructions
	below; these steps should be ignored on other versions.</p>
	<ol>
	<li>Create a local service account and group of <code>slurmrestd</code> to
	run the slurmrestd daemon. To prevent privilege escalation, the user account
	should be:
	<ul>
	<li>Not root or SlurmUser</li>
	<li>Not used by real users or for any other functions</li>
	<li>Not granted any special permissions</li>
	</ul>
	<pre>sudo useradd -M -r -s /usr/sbin/nologin -U slurmrestd</pre></li>
	<li>If needed, sites can run <code>systemctl edit slurmrestd</code> to
	override the default User and Group to the site specific user and group:
	<pre>[Service]
	User=slurmrestd
	Group=slurmrestd</pre></li>
	</ul>
	</li>
	<li>(Slurm <b>24.05</b> and newer) Optional: Customize the socket for
	slurmrestd. By default it will listen only on TCP port 6820. You may change
	this behavior by changing <code>SLURMRESTD_LISTEN</code>
	(see <a href="#customization">Customizing slurmrestd.service</a>).</li>
	<li>(Slurm <b>23.11</b> and earlier) Configure the socket for slurmrestd. This
	may be accomplished by creating/changing permissions on the parent directory
	and/or changing the path to the socket in the service file.
	<ul>
	<li><b>Permissions</b>: The user running the service must have write+execute
	permissions on the directory that will contain the UNIX socket</li>
	<li><b>Changing socket</b>: On Slurm 23.11, the way to change or disable the
	socket is to modify the 'ExecStart' line of the service
	<ol>
	<li>Run <code>systemctl edit slurmrestd</code></li>
	<li>Add the following contents to the <code>[Service]</code> section:
	<pre>
	ExecStart=
	ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS
	Environment=SLURMRESTD_LISTEN=:6820
	</pre></li>
	<li>Adjust the assignment of SLURMRESTD_LISTEN to contain the socket(s) you want
	the daemon to listen on.</li>
	<li>After a future upgrade to Slurm 24.05+, the 'ExecStart' overrides will be
	unnecessary but will not conflict with the newer version.</li>
	</ol>
	</li>
	</ul>
	</li>
	</ol>

	<h3 id="customization">Customizing slurmrestd.service
	<a class="slurm_link" href="#customization"></a></h3>

	<p>The Slurm 24.05 release changes the operation of
	the default service file and may break existing overrides. If you have
	overridden <code>ExecStart=</code> to contain any TCP/UNIX sockets directly, it
	will cause the service to fail if it duplicates any sockets contained in
	SLURMRESTD_LISTEN. These overrides will need to be changed after upgrading.</p>

	<p>The default <code>slurmrestd.service</code> file has two intended ways of
	customizing its operation:</p>

	<ol>
	<li><b>Environment files</b>:
	The service will read environment variables from two files:
	<code>/etc/default/slurmrestd</code> and <code>/etc/sysconfig/slurmrestd</code>.
	You may set any environment variables recognized by
	<a href="slurmrestd.html#SECTION_ENVIRONMENT-VARIABLES">slurmrestd</a>,
	but the following are particularly relevant:
	<ul>
	<li><b>SLURMRESTD_OPTIONS</b>: CLI options to add to the slurmrestd command
	(see <a href="slurmrestd.html">slurmrestd</a>)</li>
	<li><b>SLURMRESTD_LISTEN</b>: Comma-delimited list of host:port pairs or
	unix:$SOCKET_PATH sockets to listen on
	<br><b>NOTE</b>: If this duplicates what is already set in the
	'ExecStart' line in the service file, it will fail. Starting in Slurm 24.05,
	the default service file contains no sockets in 'ExecStart' and fully relies on
	this variable to contain the desired sockets.</li>
	</ul>
	</li>
	<li><b>Service editing</b>: Systemd has a built in way to edit services
	by running <code>systemctl edit slurmrestd</code>.
	<ul>
	<li>This will create an override file in '/etc/systemd/' containing directives
	that will add to or replace directives in the default unit in '/lib/systemd/'.
	</li>
	<li>The override file must have the appropriate section declaration(s)
	for the directives you use (e.g., <code>[Service]</code>).</li>
	<li>Environment variables may be set with <code>Environment=NAME=value</code>
	(refer to systemd documentation for more details)</li>
	<li>Changes may be reverted by running <code>systemctl revert slurmrestd</code>
	</li>
	</ul>
	</li>
	</ol>

	<h2 id="basic_usage">Basic Usage
	<a class="slurm_link" href="#basic_usage"></a></h2>

	<ol>
	<li>Find the latest supported API version
	<pre>slurmrestd -d list</pre></li>
	<li>Get an authentication token for JWT
	<pre>unset SLURM_JWT; export $(scontrol token)</pre>
	<ul>
	<li>This ensures an old token doesn't prevent a new one from being issued</li>
	<li>By default, tokens will expire after 1800 seconds (30 minutes).
	Add <code>lifespan=SECONDS</code> to the 'scontrol' command to change this.</li>
	</ul>
	</li>
	<li>Run a basic curl command to hit the API when listening on a TCP host:port
	<pre>curl -s -o "/tmp/curl.log" -k -vvvv \
	-H X-SLURM-USER-TOKEN:$SLURM_JWT \
	-X GET 'http://<server>:<port>/slurm/v0.0.<api-version>/diag'
	</pre>
	<ul>
	<li>Replace the <b>server</b>, <b>port</b>, and <b>api-version</b>
	with the appropriate values.</li>
	<li>Examine the output to ensure the response was <b>200 OK</b>,
	and examine <b>/tmp/curl.log</b> for a valid JSON response.</li>
	<li>Try other endpoints described in the <a href="rest_api.html">API Methods
	and Models</a>. Change <b>GET</b> to the correct method for the endpoint.</li>
	</ul>
	</li>
	<li>Alternate command to use the UNIX socket instead
	<pre>
	curl -s -o "/tmp/curl.log" -k -vvvv \
	-H X-SLURM-USER-TOKEN:$SLURM_JWT \
	--unix-socket /path/to/slurmrestd.socket \
	'http://<server>/slurm/v0.0.<api-version>/diag'
	</pre>
	<ul>
	<li>Replace the <b>path</b>, <b>server</b>, and <b>api-version</b>
	with the appropriate values.</li>
	<li>Examine the output to ensure the response was <b>200 OK</b>,
	and examine <b>/tmp/curl.log</b> for a valid JSON response.</li>
	</ul>
	</li>
	</ol>

	<h3 id="tokens">Token management
	<a class="slurm_link" href="#tokens"></a></h3>

	<p>This guide provides a simple overview using <code>scontrol</code> to
	obtain tokens. This is a basic introductory approach that in many cases
	should be disabled in favor of more sophisticated token management.
	Refer to the <a href="jwt.html">JWT page</a> for more details.</p>

	<h2 id="advanced_usage">Advanced Usage
	<a class="slurm_link" href="#advanced_usage"></a></h2>

	<p>Information about ways to further customize and configure slurmrestd,
	including authentication methods, run modes, plugins, high availability, proxies,
	and Python client is found on the <a href="rest.html">REST API Details</a> page.</p>

	<h2 id="common_issues">Common Issues
	<a class="slurm_link" href="#common_issues"></a></h2>

	<p>In general, look out for these things:</p>

	<ol>
	<li>Validity of authentication token in <code>SLURM_JWT</code></li>
	<li>Hostname and port number</li>
	<li>API version and endpoint</li>
	<li>Log output of slurmrestd</li>
	</ol>

	<h3 id="bind_socket">Unable to bind socket
	<a class="slurm_link" href="#bind_socket"></a></h3>

	<p>This may be due to a permissions issue while attempting to set up the socket.
	Check the log output from slurmrestd for the path to the socket.
	Ensure that the user running the slurmrestd service has permissions to the
	parent directory of the configured socket path, or change/remove the socket path
	as <a href="#systemd">described above</a>.</p>

	<p>If it says "<b>Address already in use</b>", check the command being run
	and the contents of "SLURMRESTD_LISTEN" for duplicates of the same TCP or UNIX
	socket.</p>

	<h3 id="connection_refused">Connection refused
	<a class="slurm_link" href="#connection_refused"></a></h3>

	<p>Verify that slurmrestd is running and listening on the port you are
	attempting to connect to.</p>

	<h3 id="authentication_error">Protocol authentication error (HTTP 500)
	<a class="slurm_link" href="#authentication_error"></a></h3>

	<p>One common authentication problem is an expired token. Request a new one:
	<pre>unset SLURM_JWT; export $(scontrol token)</pre></p>

	<p>This solution also applies to an HTTP 401 error caused by no authentication
	token being sent at all. This may appear in the slurmrestd logs as
	"Authentication does not apply to request."</p>

	<p>Otherwise, consult the logs on the <b>slurmctld</b> and <b>slurmdbd</b>.</p>

	<h3 id="invalid_url">Unable to find requested URL (HTTP 404)
	<a class="slurm_link" href="#invalid_url"></a></h3>

	<p>Check the <a href="rest_api.html">API Methods and Models</a> page to ensure
	you're using a valid URL and the correct method for it. Pay attention to the
	path as there are different endpoints for <b>slurm</b> and <b>slurmdbd</b>.</p>

	<h3 id="rejected_token">Rejecting thread config token (HTTP 401)
	<a class="slurm_link" href="#rejected_token"></a></h3>

	<p>Check that slurmrestd has loaded the <b>auth/jwt</b> plugin.
	You should see a debug message like this:
	<pre>slurmrestd: debug: auth/jwt: init: JWT authentication plugin loaded</pre>
	If it didn't load jwt, run this in the terminal you're using for slurmrestd:
	<pre>export SLURM_JWT=daemon</pre>
	</p>

	<h3 id="unexpected_character">Unexpected URL character (HTTP 400)
	<a class="slurm_link" href="#unexpected_character"></a></h3>

	<p>Check the request URL and slurmrestd logs for characters that may be causing
	the URL to be parsed incorrectly. Use the appropriate URL encoding sequence in
	place of the problematic character (e.g., <b>/</b> = <b>%2F</b>).
	<pre>
	... -X GET "localhost:8080/slurmdb/v0.0.40/jobs?submit_time=02/28/24"
	### 400 BAD REQUEST
	... -X GET "localhost:8080/slurmdb/v0.0.40/jobs?submit_time=02%2F28%2F24"
	### 200 OK
	</pre>
	</p>

	<h3 id="slurm_commands">Other slurm commands not working
	<a class="slurm_link" href="#slurm_commands"></a></h3>

	<p>If SLURM_JWT is set, other slurm commands will attempt to use JWT
	authentication, causing failures. This can be fixed by clearing the variable:
	<pre>unset SLURM_JWT</pre></p>

	<!--#include virtual="footer.txt"-->