| <!--#include virtual="header.txt"--> |
| |
| <h1>REST API Quick Start Guide</h1> |
| |
| <p>Slurm provides a <a href="rest.html">REST API</a> through the slurmrestd |
| daemon, using <a href="jwt.html">JSON Web Tokens</a> for authentication. |
| This page provides a brief tutorial for setting up these components.</p> |
| |
| <p> |
| See also: |
| <ul> |
| <li><a href="rest.html">REST API Details</a></li> |
| <li><a href="rest_api.html">REST API Methods and Models</a></li> |
| <li><a href="slurmrestd.html">slurmrestd man page</a></li> |
| </ul> |
| </p> |
| |
| <h2 id="contents">Contents<a class="slurm_link" href="#contents"></a></h2> |
| <ul> |
| <li><a href="#prereq">Prerequisites</a></li> |
| <li><a href="#quick_start">Quick Start</a> |
| <ul> |
| <li><a href="#systemd">Running with systemd</a></li> |
| <li><a href="#customization">Customizing slurmrestd.service</a></li> |
| </ul> |
| </li> |
| <li><a href="#basic_usage">Basic Usage</a> |
| <ul> |
| <li><a href="#tokens">Token management</a></li> |
| </ul> |
| </li> |
| <li><a href="#common_issues">Common Issues</a> |
| <ul> |
| <li><a href="#bind_socket">Unable to bind socket</a></li> |
| <li><a href="#connection_refused">Connection refused</a></li> |
| <li><a href="#authentication_error">Protocol authentication error (HTTP 500) |
| </a></li> |
| <li><a href="#invalid_url">Unable to find requested URL (HTTP 404)</a></li> |
| <li><a href="#rejected_token">Rejecting thread config token (HTTP 401) |
| </a></li> |
| <li><a href="#unexpected_character">Unexpected URL character (HTTP 400) |
| </a></li> |
| <li><a href="#slurm_commands">Other Slurm commands not working</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h2 id="prereq">Prerequisites |
| <a class="slurm_link" href="#prereq"></a></h2> |
| <p>The following development libraries are required at compile time |
| in order for slurmrestd to be compiled (minimum versions are on the related |
| software page linked below):</p> |
| <ul> |
| <li><a href="related_software.html#httpparser">HTTP Parser</a> |
| </li> |
| <li><a href="related_software.html#yaml">LibYAML</a> (optional) |
| </li> |
| <li><a href="related_software.html#json">JSON-C</a></li> |
| <li><a href="related_software.html#jwt">JWT Authentication</a> |
| (optional, used in this guide)</li> |
| </ul> |
| |
| <p>Additionally, it is highly recommended that you have |
| <a href="slurmdbd.html">SlurmDBD</a> set up for |
| <a href="accounting.html">accounting</a>. Without slurmdbd, slurmrestd may fail |
| when loading some plugins (use |
| <a href="slurmrestd.html#OPT_-s-<OpenAPI-plugins-to-load>">-s</a> to specify |
| which to load) or when attempting JWT authentication.</p> |
| |
| <h2 id="quick_start">Quick Start |
| <a class="slurm_link" href="#quick_start"></a></h2> |
| |
| <p>This may be done on a dedicated REST API machine or |
| your existing 'slurmctld' machine, depending on demand. |
| If you have multiple clusters, you will need a unique instance of |
| <b>slurmrestd</b> for each cluster.</p> |
| |
| <ol> |
| <li>Install components for slurmrestd |
| <ul> |
| <li>DEB: <code>slurm-smd slurm-smd-slurmrestd</code></li> |
| <li>RPM: <code>slurm slurm-slurmrestd</code> (requires |
| <code>--with slurmrestd</code> at build time)</li> |
| </ul></li> |
| <li>Set up <a href="jwt.html">JSON Web Tokens</a> for authentication</li> |
| <li>Ensure <code>/etc/slurm/slurm.conf</code> is present and correct for your |
| cluster (see <a href="quickstart_admin.html">Quick Start Admin Guide</a> and |
| <a href="slurm.conf.html">slurm.conf man page</a>)</li> |
| <li>Run <b>slurmrestd</b> (see <a href="#systemd">below</a> for systemd |
| instructions) on your preferred [HOST]:PORT combination |
| (':6820' is the default for production) |
| <pre>export SLURM_JWT=daemon |
| export SLURMRESTD_DEBUG=debug |
| slurmrestd <host>:<port> |
| </pre> |
| Adjust SLURMRESTD_DEBUG to the desired level of output (as described on the |
| <a href="slurmrestd.html#OPT_SLURMRESTD_DEBUG">man page</a>)</li> |
| </ol> |
| |
| <h3 id="systemd">Running with systemd |
| <a class="slurm_link" href="#systemd"></a></h3> |
| |
| <p>Slurm ships with a <b>slurmrestd</b> service unit for systemd, |
| however, it might require some additional setup to run properly. |
| This section assumes you have either installed slurmrestd using DEB/RPM packages |
| or built it manually such that the files are in the same places. |
| <br><b>Note</b> the versions associated with certain steps in the instructions |
| below; these steps should be ignored on other versions.</p> |
| <ol> |
| <li>Create a local service account and group of <code>slurmrestd</code> to |
| run the slurmrestd daemon. To prevent privilege escalation, the user account |
| should be: |
| <ul> |
| <li>Not root or SlurmUser</li> |
| <li>Not used by real users or for any other functions</li> |
| <li>Not granted any special permissions</li> |
| </ul> |
| <pre>sudo useradd -M -r -s /usr/sbin/nologin -U slurmrestd</pre></li> |
| <li>If needed, sites can run <code>systemctl edit slurmrestd</code> to |
| override the default User and Group to the site specific user and group: |
| <pre>[Service] |
| User=slurmrestd |
| Group=slurmrestd</pre></li> |
| </ul> |
| </li> |
| <li>(Slurm <b>24.05</b> and newer) Optional: Customize the socket for |
| slurmrestd. By default it will listen only on TCP port 6820. You may change |
| this behavior by changing <code>SLURMRESTD_LISTEN</code> |
| (see <a href="#customization">Customizing slurmrestd.service</a>).</li> |
| <li>(Slurm <b>23.11</b> and earlier) Configure the socket for slurmrestd. This |
| may be accomplished by creating/changing permissions on the parent directory |
| and/or changing the path to the socket in the service file. |
| <ul> |
| <li><b>Permissions</b>: The user running the service must have write+execute |
| permissions on the directory that will contain the UNIX socket</li> |
| <li><b>Changing socket</b>: On Slurm 23.11, the way to change or disable the |
| socket is to modify the 'ExecStart' line of the service |
| <ol> |
| <li>Run <code>systemctl edit slurmrestd</code></li> |
| <li>Add the following contents to the <code>[Service]</code> section: |
| <pre> |
| ExecStart= |
| ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS |
| Environment=SLURMRESTD_LISTEN=:6820 |
| </pre></li> |
| <li>Adjust the assignment of SLURMRESTD_LISTEN to contain the socket(s) you want |
| the daemon to listen on.</li> |
| <li>After a future upgrade to Slurm 24.05+, the 'ExecStart' overrides will be |
| unnecessary but will not conflict with the newer version.</li> |
| </ol> |
| </li> |
| </ul> |
| </li> |
| </ol> |
| |
| <h3 id="customization">Customizing slurmrestd.service |
| <a class="slurm_link" href="#customization"></a></h3> |
| |
| <p>The Slurm 24.05 release changes the operation of |
| the default service file and may break existing overrides. If you have |
| overridden <code>ExecStart=</code> to contain any TCP/UNIX sockets directly, it |
| will cause the service to fail if it duplicates any sockets contained in |
| SLURMRESTD_LISTEN. These overrides will need to be changed after upgrading.</p> |
| |
| <p>The default <code>slurmrestd.service</code> file has two intended ways of |
| customizing its operation:</p> |
| |
| <ol> |
| <li><b>Environment files</b>: |
| The service will read environment variables from two files: |
| <code>/etc/default/slurmrestd</code> and <code>/etc/sysconfig/slurmrestd</code>. |
| You may set any environment variables recognized by |
| <a href="slurmrestd.html#SECTION_ENVIRONMENT-VARIABLES">slurmrestd</a>, |
| but the following are particularly relevant: |
| <ul> |
| <li><b>SLURMRESTD_OPTIONS</b>: CLI options to add to the slurmrestd command |
| (see <a href="slurmrestd.html">slurmrestd</a>)</li> |
| <li><b>SLURMRESTD_LISTEN</b>: Comma-delimited list of host:port pairs or |
| unix:$SOCKET_PATH sockets to listen on |
| <br><b>NOTE</b>: If this duplicates what is already set in the |
| 'ExecStart' line in the service file, it will fail. Starting in Slurm 24.05, |
| the default service file contains no sockets in 'ExecStart' and fully relies on |
| this variable to contain the desired sockets.</li> |
| </ul> |
| </li> |
| <li><b>Service editing</b>: Systemd has a built in way to edit services |
| by running <code>systemctl edit slurmrestd</code>. |
| <ul> |
| <li>This will create an override file in '/etc/systemd/' containing directives |
| that will add to or replace directives in the default unit in '/lib/systemd/'. |
| </li> |
| <li>The override file must have the appropriate section declaration(s) |
| for the directives you use (e.g., <code>[Service]</code>).</li> |
| <li>Environment variables may be set with <code>Environment=NAME=value</code> |
| (refer to systemd documentation for more details)</li> |
| <li>Changes may be reverted by running <code>systemctl revert slurmrestd</code> |
| </li> |
| </ul> |
| </li> |
| </ol> |
| |
| <h2 id="basic_usage">Basic Usage |
| <a class="slurm_link" href="#basic_usage"></a></h2> |
| |
| <ol> |
| <li>Find the latest supported API version |
| <pre>slurmrestd -d list</pre></li> |
| <li>Get an authentication token for JWT |
| <pre>unset SLURM_JWT; export $(scontrol token)</pre> |
| <ul> |
| <li>This ensures an old token doesn't prevent a new one from being issued</li> |
| <li>By default, tokens will expire after 1800 seconds (30 minutes). |
| Add <code>lifespan=SECONDS</code> to the 'scontrol' command to change this.</li> |
| </ul> |
| </li> |
| <li>Run a basic curl command to hit the API when listening on a TCP host:port |
| <pre>curl -s -o "/tmp/curl.log" -k -vvvv \ |
| -H X-SLURM-USER-TOKEN:$SLURM_JWT \ |
| -X GET 'http://<server>:<port>/slurm/v0.0.<api-version>/diag' |
| </pre> |
| <ul> |
| <li>Replace the <b>server</b>, <b>port</b>, and <b>api-version</b> |
| with the appropriate values.</li> |
| <li>Examine the output to ensure the response was <b>200 OK</b>, |
| and examine <b>/tmp/curl.log</b> for a valid JSON response.</li> |
| <li>Try other endpoints described in the <a href="rest_api.html">API Methods |
| and Models</a>. Change <b>GET</b> to the correct method for the endpoint.</li> |
| </ul> |
| </li> |
| <li>Alternate command to use the UNIX socket instead |
| <pre> |
| curl -s -o "/tmp/curl.log" -k -vvvv \ |
| -H X-SLURM-USER-TOKEN:$SLURM_JWT \ |
| --unix-socket /path/to/slurmrestd.socket \ |
| 'http://<server>/slurm/v0.0.<api-version>/diag' |
| </pre> |
| <ul> |
| <li>Replace the <b>path</b>, <b>server</b>, and <b>api-version</b> |
| with the appropriate values.</li> |
| <li>Examine the output to ensure the response was <b>200 OK</b>, |
| and examine <b>/tmp/curl.log</b> for a valid JSON response.</li> |
| </ul> |
| </li> |
| </ol> |
| |
| <h3 id="tokens">Token management |
| <a class="slurm_link" href="#tokens"></a></h3> |
| |
| <p>This guide provides a simple overview using <code>scontrol</code> to |
| obtain tokens. This is a basic introductory approach that in many cases |
| should be disabled in favor of more sophisticated token management. |
| Refer to the <a href="jwt.html">JWT page</a> for more details.</p> |
| |
| <h2 id="advanced_usage">Advanced Usage |
| <a class="slurm_link" href="#advanced_usage"></a></h2> |
| |
| <p>Information about ways to further customize and configure slurmrestd, |
| including authentication methods, run modes, plugins, high availability, and |
| proxies, is found on the <a href="rest.html">REST API Details</a> page.</p> |
| |
| <h2 id="common_issues">Common Issues |
| <a class="slurm_link" href="#common_issues"></a></h2> |
| |
| <p>In general, look out for these things:</p> |
| |
| <ol> |
| <li>Validity of authentication token in <code>SLURM_JWT</code></li> |
| <li>Hostname and port number</li> |
| <li>API version and endpoint</li> |
| <li>Log output of slurmrestd</li> |
| </ol> |
| |
| <h3 id="bind_socket">Unable to bind socket |
| <a class="slurm_link" href="#bind_socket"></a></h3> |
| |
| <p>This may be due to a permissions issue while attempting to set up the socket. |
| Check the log output from slurmrestd for the path to the socket. |
| Ensure that the user running the slurmrestd service has permissions to the |
| parent directory of the configured socket path, or change/remove the socket path |
| as <a href="#systemd">described above</a>.</p> |
| |
| <p>If it says "<b>Address already in use</b>", check the command being run |
| and the contents of "SLURMRESTD_LISTEN" for duplicates of the same TCP or UNIX |
| socket.</p> |
| |
| <h3 id="connection_refused">Connection refused |
| <a class="slurm_link" href="#connection_refused"></a></h3> |
| |
| <p>Verify that slurmrestd is running and listening on the port you are |
| attempting to connect to.</p> |
| |
| <h3 id="authentication_error">Protocol authentication error (HTTP 500) |
| <a class="slurm_link" href="#authentication_error"></a></h3> |
| |
| <p>One common authentication problem is an expired token. Request a new one: |
| <pre>unset SLURM_JWT; export $(scontrol token)</pre></p> |
| |
| <p>This solution also applies to an HTTP 401 error caused by no authentication |
| token being sent at all. This may appear in the slurmrestd logs as |
| "Authentication does not apply to request."</p> |
| |
| <p>Otherwise, consult the logs on the <b>slurmctld</b> and <b>slurmdbd</b>.</p> |
| |
| <h3 id="invalid_url">Unable to find requested URL (HTTP 404) |
| <a class="slurm_link" href="#invalid_url"></a></h3> |
| |
| <p>Check the <a href="rest_api.html">API Methods and Models</a> page to ensure |
| you're using a valid URL and the correct method for it. Pay attention to the |
| path as there are different endpoints for <b>slurm</b> and <b>slurmdbd</b>.</p> |
| |
| <h3 id="rejected_token">Rejecting thread config token (HTTP 401) |
| <a class="slurm_link" href="#rejected_token"></a></h3> |
| |
| <p>Check that slurmrestd has loaded the <b>auth/jwt</b> plugin. |
| You should see a debug message like this: |
| <pre>slurmrestd: debug: auth/jwt: init: JWT authentication plugin loaded</pre> |
| If it didn't load jwt, run this in the terminal you're using for slurmrestd: |
| <pre>export SLURM_JWT=daemon</pre> |
| </p> |
| |
| <h3 id="unexpected_character">Unexpected URL character (HTTP 400) |
| <a class="slurm_link" href="#unexpected_character"></a></h3> |
| |
| <p>Check the request URL and slurmrestd logs for characters that may be causing |
| the URL to be parsed incorrectly. Use the appropriate URL encoding sequence in |
| place of the problematic character (e.g., <b>/</b> = <b>%2F</b>). |
| <pre> |
| ... -X GET "localhost:8080/slurmdb/v0.0.40/jobs?submit_time=02/28/24" |
| ### 400 BAD REQUEST |
| ... -X GET "localhost:8080/slurmdb/v0.0.40/jobs?submit_time=02%2F28%2F24" |
| ### 200 OK |
| </pre> |
| </p> |
| |
| <h3 id="slurm_commands">Other slurm commands not working |
| <a class="slurm_link" href="#slurm_commands"></a></h3> |
| |
| <p>If SLURM_JWT is set, other slurm commands will attempt to use JWT |
| authentication, causing failures. This can be fixed by clearing the variable: |
| <pre>unset SLURM_JWT</pre></p> |
| |
| <p style="text-align:center;">Last modified 16 May 2025</p> |
| |
| <!--#include virtual="footer.txt"--> |