RELEASE_NOTES - SchedMD/slurm - Git at Google

 RELEASE NOTES FOR SLURM VERSION 18.08
 16 August 2018

 IMPORTANT NOTES:
 If using the slurmdbd (Slurm DataBase Daemon) you must update this first.

 NOTE: If using a backup DBD you must start the primary first to do any
 database conversion, the backup will not start until this has happened.

 The 18.08 slurmdbd will work with Slurm daemons of version 17.02 and above.
 You will not need to update all clusters at the same time, but it is very
 important to update slurmdbd first and having it running before updating
 any other clusters making use of it.

 Slurm can be upgraded from version 17.02 or 17.11 to version 18.08 without loss
 of jobs or other state information. Upgrading directly from an earlier version
 of Slurm will result in loss of state information.

 If using SPANK plugins that use the Slurm APIs, they should be recompiled when
 upgrading Slurm to a new major release.

 NOTE FOR THOSE UPGRADING SLURMDBD: The database conversion process from
       SlurmDBD 16.05 or 17.02 may not work properly with MySQL 5.1 or 5.5
       (as was the default version for RHEL 6). Upgrading to a newer version of
       MariaDB or MySQL is strongly encouraged to prevent this problem.

 NOTE: The slurmctld is now set to fatal if there are any problems with
       any state files.  To avoid this use the new '-i' flag.

 NOTE: systemd services files are installed automatically, but not enabled.
       You will need to manually enable them on the appropriate systems:
       - Controller: systemctl enable slurmctld
       - Database: systemctl enable slurmdbd
       - Compute Nodes: systemctl enable slurmd

 NOTE: If you are upgrading with any jobs from 14.03 or earlier
       (i.e. quick upgrade from 14.03 -> 15.08 -> 17.02) you will need
       to wait until after those jobs are gone before you upgrade to 17.02
       or 17.11 or 18.08.

 NOTE: If you interact with any memory values in a job_submit plugin, you will
       need to test against NO_VAL64 instead of NO_VAL, and change your printf
       format as well.

 NOTE: SPANK options handled in local and allocator contexts should be able to
       handle being called multiple times. An option could be set multiple times
       through environment variables and command line options. Environment
       variables are processed first.

 NOTE: IBM BlueGene/Q support has been removed.

 NOTE: Cray/ALPS is deprecated and will be removed in the 19.05 release.
       You must add the --enable-deprecated option to configure to build
       for this target.

 NOTE: Built-in BLCR support is deprecated, no longer built automatically, and
       will be removed in 19.05. You must add --with-blcr and --enable-deprecated
       options to configure to build this plugin.

 NOTE: srun will now only read in the environment variables SLURM_JOB_NODES and
       SLURM_JOB_NODELIST instead of SLURM_NNODES and SLURM_NODELIST.  These
       latter variables have been obsolete for some time please update any
       scripts still using them.

 NOTE: squeue will now print "UID" as the header for the User ID field;
       corresponding to the %U type specification.

 NOTE: slurmd and slurmctld will now fatal if two incompatible mechanisms for
       enforcing memory limits are set. This makes incompatible the use of
       task/cgroup memory limit enforcing (Constrain[RAM|Swap]Space=yes) with
       MemLimitEnforce=yes or with JobAcctGatherParams=OverMemoryKill, which
       could cause problems when a task is killed by one of them while the other
       is at the same time managing that task. The NoOverMemoryKill setting has
       been deprecated in favour of OverMemoryKill, since now the default is
       *NOT* to have any memory enforcement mechanism.

 HIGHLIGHTS
 ==========
  -- Add support for parenthesis in an advanced reservation or job constraint
     specification to group like options together. For example:
     --constraint="[(knl&snc4&flat)*4&haswell*1]" might be used to specify that
     four nodes with the features "knl", "snc4" and "flat" plus one node with
     the feature "haswell" are required. Nodes with those features currently
     available will be favored.
  -- Heterogeneous job steps allocations supported with
     * Open MPI (with Slurm's PMI2 and PMIx plugins) and
     * Intel MPI (with Slurm's PMI2 plugin)
     * No support for Cray systems at this time.
  -- Disable local PTY output processing when using 'srun --unbuffered'. This
     prevents the PTY subsystem from inserting extraneous \r characters into
     the output stream.
  -- Changed slurmd.service file to include "remote-fs.target" in "After" option.
  -- The default AuthType for slurmdbd is now "auth/munge".

 CONFIGURATION FILE CHANGES (see man appropriate man page for details)
 =====================================================================
  -- Add node and partition configuration options of "CpuBind" in slurm.conf to
     control default task binding. Modify the scontrol to report and modify
     these parameters.
  -- Add "NumaCpuBind" option to knl.conf file to automatically change a node's
     CpuBind parameter based upon changes to a node's NUMA mode.
  -- Add "ValidateMode" configuration parameter to knl.conf for static
     MCDRAM/NUMA configurations.
  -- Add "NodeRebootWeight" option to knl.conf file to modiify a node's
     scheduling weight when rebooting is required to satify a job's constraints.
  -- Add "GetSysStatus" option to burst_buffer.conf file. For burst_buffer/cray
     this would indicate the location of the "dwstat" command.
  -- Add "SetExecHost" flag to burst_buffer.conf to enable access from login node
     for interactive jobs
  -- Configuration parameters "ControlMachine", "ControlAddr", "BackupController"
     and "BackupAddr" in slurm.conf replaced by an ordered list of
     "SlurmctldHost" records with the optional address appended to the name
     enclosed in parenthesis. For example: "SlurmctldHost=head(12.34.56.78)".
     An arbitrary number of backup servers can be configured.
  -- Add "SlurmctldPrimaryOnProg" and "SlurmctldPrimaryOffProg" options to
     slurm.conf to run scripts when primary/backup slurmctld server changes.
  -- Add "ResumeFailProgram" slurm.conf option to specify a program that is
     called when a node fails to respond by the configured "ResumeTimeout".
  -- Remove support for "ChosLoc" configuration parameter.
  -- Add SlurmctldParameters option allow_user_triggers to enable user-defined
     triggers. User defined triggers are now disabled by default.
  -- ConstrainKmemSpace is now disabled by default. In Linux kernel version 3,
     early versions of kernel version 4, and RedHat/CentOS 6 and 7, using
     memory.kmem.limit_in_bytes leaks slab caches, which eventually causes the
     machine to be unable to create new cgroups.
  -- SchedulerParameters' "whole_pack" option has been renamed to "whole_hetjob",
     although "whole_pack" will still be supported for backward compatibility.
  -- Cray: Add "CheckGhalQuiesce" to "CommunicationParameters" in slurm.conf.
  -- Remove requirement to have cgroup_allowed_devices_file.conf in order to
     constrain devices. By default all devices are allowed and GRES, that are
     associated with a device file, that are not requested are restricted.

 COMMAND CHANGES (see man pages for details)
 ===========================================
  -- Add sbatch "--batch" option to identify features required on batch node.
     For example "sbatch --batch=haswell ...".
  -- sacct can report all information about all components of a heterogeneous
     job allocation with "--whole-hetjob=yes" option.
  -- Report cgroup and NodeFeatures plugin configuration with scontrol and
     sview commands.
  -- Add "scontrol show dwstat" command to display Cray burst buffer status.
  -- Remove the "CPUErr" value from the slurm_print_node_table() API call
     ("scontrol show node"). This was always zero for non-BlueGene systems.
  -- srun command returns the highest signal of any task.
  -- Add salloc/sbatch/srun option of "--gres-flags=disable-binding" to disable
     filtering of CPUs with respect to generic resource locality. This option is
     currently required to use more CPUs than are bound to a GRES (i.e. if a GPU
     is bound to the CPUs on one socket, but resources on more than one socket
     are required to run the job).
  -- Add name of partition used to output of "srun --test-only ...". This is
     valuable for jobs submitted to multiple partitions.
  -- Add job reason "ReqNodeNotAvail, reserved for maintenance".
  -- "scontrol reboot" enhancements:
     - Add ability to specify node reason
     - Add ability to specify node state after reboot completion
  -- sdiag command enhancements:
     - Report outgoing message queue contents
     - Report pending job count
  -- Add scontrol ability to create/update "TRESBillingWeights".
  -- Remove "AdminComment +=" syntax from "scontrol update job".
  -- Add sacctmgr options to prevent/manage job queue stuffing:
     - GrpJobsAccrue=<max_jobs>
       Maximum number of pending jobs in aggregate able to accrue age priority
       for this association and all associations which are children of this
       association. To clear a previously set value use the modify command with
       a new value of -1.
     - MaxJobsAccrue=<max_jobs>
       Maximum number of pending jobs able to accrue age priority at any given
       time for the given association. This is overridden if set directly on a
       user. Default is the cluster's limit. To clear a previously set value use
       the modify command with a new value of -1.
     - MinPrioThreshold
       Minimum priority required to reserve resources when scheduling.

 OTHER CHANGES
 =============
  -- Add new job state of SO/STAGE_OUT indicating that burst buffer stage-out
     operation is in progress.
  -- Add new burst buffer state of "teardown-fail".
  -- Burst buffer errors logged to new job "SystemComment" field.
  -- Enable jobs with zero node count for creation and/or deletion of persistent
     burst buffers.
  -- Avoid terminating other processes in a task group when any task is core
     dumping to avoid incomplete OpenMP core files.
  -- Append ", with requeued tasks" to end of job array "end" email when any task
     is requeued. This is a hint to use "sacct --duplicates" to see all job
     accounting information.
  -- Consider nodes in "Resuming" state as available for backfill future
     scheduling and do not replace in advanced reservations.
  -- Explicitly shutdown the slurmd daemon when reboot requested.
  -- Calculate TRES billing values at job submission to enforce QOS DenyOnLimit
     configuration.
  -- Add acct_gather_profile/influxdb plugin to store job profiling information
     to InfluxDB rather than HDF5.
	RELEASE NOTES FOR SLURM VERSION 18.08
	16 August 2018

	IMPORTANT NOTES:
	If using the slurmdbd (Slurm DataBase Daemon) you must update this first.

	NOTE: If using a backup DBD you must start the primary first to do any
	database conversion, the backup will not start until this has happened.

	The 18.08 slurmdbd will work with Slurm daemons of version 17.02 and above.
	You will not need to update all clusters at the same time, but it is very
	important to update slurmdbd first and having it running before updating
	any other clusters making use of it.

	Slurm can be upgraded from version 17.02 or 17.11 to version 18.08 without loss
	of jobs or other state information. Upgrading directly from an earlier version
	of Slurm will result in loss of state information.

	If using SPANK plugins that use the Slurm APIs, they should be recompiled when
	upgrading Slurm to a new major release.

	NOTE FOR THOSE UPGRADING SLURMDBD: The database conversion process from
	SlurmDBD 16.05 or 17.02 may not work properly with MySQL 5.1 or 5.5
	(as was the default version for RHEL 6). Upgrading to a newer version of
	MariaDB or MySQL is strongly encouraged to prevent this problem.

	NOTE: The slurmctld is now set to fatal if there are any problems with
	any state files. To avoid this use the new '-i' flag.

	NOTE: systemd services files are installed automatically, but not enabled.
	You will need to manually enable them on the appropriate systems:
	- Controller: systemctl enable slurmctld
	- Database: systemctl enable slurmdbd
	- Compute Nodes: systemctl enable slurmd

	NOTE: If you are upgrading with any jobs from 14.03 or earlier
	(i.e. quick upgrade from 14.03 -> 15.08 -> 17.02) you will need
	to wait until after those jobs are gone before you upgrade to 17.02
	or 17.11 or 18.08.

	NOTE: If you interact with any memory values in a job_submit plugin, you will
	need to test against NO_VAL64 instead of NO_VAL, and change your printf
	format as well.

	NOTE: SPANK options handled in local and allocator contexts should be able to
	handle being called multiple times. An option could be set multiple times
	through environment variables and command line options. Environment
	variables are processed first.

	NOTE: IBM BlueGene/Q support has been removed.

	NOTE: Cray/ALPS is deprecated and will be removed in the 19.05 release.
	You must add the --enable-deprecated option to configure to build
	for this target.

	NOTE: Built-in BLCR support is deprecated, no longer built automatically, and
	will be removed in 19.05. You must add --with-blcr and --enable-deprecated
	options to configure to build this plugin.

	NOTE: srun will now only read in the environment variables SLURM_JOB_NODES and
	SLURM_JOB_NODELIST instead of SLURM_NNODES and SLURM_NODELIST. These
	latter variables have been obsolete for some time please update any
	scripts still using them.

	NOTE: squeue will now print "UID" as the header for the User ID field;
	corresponding to the %U type specification.

	NOTE: slurmd and slurmctld will now fatal if two incompatible mechanisms for
	enforcing memory limits are set. This makes incompatible the use of
	task/cgroup memory limit enforcing (Constrain[RAM\|Swap]Space=yes) with
	MemLimitEnforce=yes or with JobAcctGatherParams=OverMemoryKill, which
	could cause problems when a task is killed by one of them while the other
	is at the same time managing that task. The NoOverMemoryKill setting has
	been deprecated in favour of OverMemoryKill, since now the default is
	NOT to have any memory enforcement mechanism.

	HIGHLIGHTS
	==========
	-- Add support for parenthesis in an advanced reservation or job constraint
	specification to group like options together. For example:
	--constraint="[(knl&snc4&flat)4&haswell1]" might be used to specify that
	four nodes with the features "knl", "snc4" and "flat" plus one node with
	the feature "haswell" are required. Nodes with those features currently
	available will be favored.
	-- Heterogeneous job steps allocations supported with
	* Open MPI (with Slurm's PMI2 and PMIx plugins) and
	* Intel MPI (with Slurm's PMI2 plugin)
	* No support for Cray systems at this time.
	-- Disable local PTY output processing when using 'srun --unbuffered'. This
	prevents the PTY subsystem from inserting extraneous \r characters into
	the output stream.
	-- Changed slurmd.service file to include "remote-fs.target" in "After" option.
	-- The default AuthType for slurmdbd is now "auth/munge".

	CONFIGURATION FILE CHANGES (see man appropriate man page for details)
	=====================================================================
	-- Add node and partition configuration options of "CpuBind" in slurm.conf to
	control default task binding. Modify the scontrol to report and modify
	these parameters.
	-- Add "NumaCpuBind" option to knl.conf file to automatically change a node's
	CpuBind parameter based upon changes to a node's NUMA mode.
	-- Add "ValidateMode" configuration parameter to knl.conf for static
	MCDRAM/NUMA configurations.
	-- Add "NodeRebootWeight" option to knl.conf file to modiify a node's
	scheduling weight when rebooting is required to satify a job's constraints.
	-- Add "GetSysStatus" option to burst_buffer.conf file. For burst_buffer/cray
	this would indicate the location of the "dwstat" command.
	-- Add "SetExecHost" flag to burst_buffer.conf to enable access from login node
	for interactive jobs
	-- Configuration parameters "ControlMachine", "ControlAddr", "BackupController"
	and "BackupAddr" in slurm.conf replaced by an ordered list of
	"SlurmctldHost" records with the optional address appended to the name
	enclosed in parenthesis. For example: "SlurmctldHost=head(12.34.56.78)".
	An arbitrary number of backup servers can be configured.
	-- Add "SlurmctldPrimaryOnProg" and "SlurmctldPrimaryOffProg" options to
	slurm.conf to run scripts when primary/backup slurmctld server changes.
	-- Add "ResumeFailProgram" slurm.conf option to specify a program that is
	called when a node fails to respond by the configured "ResumeTimeout".
	-- Remove support for "ChosLoc" configuration parameter.
	-- Add SlurmctldParameters option allow_user_triggers to enable user-defined
	triggers. User defined triggers are now disabled by default.
	-- ConstrainKmemSpace is now disabled by default. In Linux kernel version 3,
	early versions of kernel version 4, and RedHat/CentOS 6 and 7, using
	memory.kmem.limit_in_bytes leaks slab caches, which eventually causes the
	machine to be unable to create new cgroups.
	-- SchedulerParameters' "whole_pack" option has been renamed to "whole_hetjob",
	although "whole_pack" will still be supported for backward compatibility.
	-- Cray: Add "CheckGhalQuiesce" to "CommunicationParameters" in slurm.conf.
	-- Remove requirement to have cgroup_allowed_devices_file.conf in order to
	constrain devices. By default all devices are allowed and GRES, that are
	associated with a device file, that are not requested are restricted.

	COMMAND CHANGES (see man pages for details)
	===========================================
	-- Add sbatch "--batch" option to identify features required on batch node.
	For example "sbatch --batch=haswell ...".
	-- sacct can report all information about all components of a heterogeneous
	job allocation with "--whole-hetjob=yes" option.
	-- Report cgroup and NodeFeatures plugin configuration with scontrol and
	sview commands.
	-- Add "scontrol show dwstat" command to display Cray burst buffer status.
	-- Remove the "CPUErr" value from the slurm_print_node_table() API call
	("scontrol show node"). This was always zero for non-BlueGene systems.
	-- srun command returns the highest signal of any task.
	-- Add salloc/sbatch/srun option of "--gres-flags=disable-binding" to disable
	filtering of CPUs with respect to generic resource locality. This option is
	currently required to use more CPUs than are bound to a GRES (i.e. if a GPU
	is bound to the CPUs on one socket, but resources on more than one socket
	are required to run the job).
	-- Add name of partition used to output of "srun --test-only ...". This is
	valuable for jobs submitted to multiple partitions.
	-- Add job reason "ReqNodeNotAvail, reserved for maintenance".
	-- "scontrol reboot" enhancements:
	- Add ability to specify node reason
	- Add ability to specify node state after reboot completion
	-- sdiag command enhancements:
	- Report outgoing message queue contents
	- Report pending job count
	-- Add scontrol ability to create/update "TRESBillingWeights".
	-- Remove "AdminComment +=" syntax from "scontrol update job".
	-- Add sacctmgr options to prevent/manage job queue stuffing:
	- GrpJobsAccrue=<max_jobs>
	Maximum number of pending jobs in aggregate able to accrue age priority
	for this association and all associations which are children of this
	association. To clear a previously set value use the modify command with
	a new value of -1.
	- MaxJobsAccrue=<max_jobs>
	Maximum number of pending jobs able to accrue age priority at any given
	time for the given association. This is overridden if set directly on a
	user. Default is the cluster's limit. To clear a previously set value use
	the modify command with a new value of -1.
	- MinPrioThreshold
	Minimum priority required to reserve resources when scheduling.

	OTHER CHANGES
	=============
	-- Add new job state of SO/STAGE_OUT indicating that burst buffer stage-out
	operation is in progress.
	-- Add new burst buffer state of "teardown-fail".
	-- Burst buffer errors logged to new job "SystemComment" field.
	-- Enable jobs with zero node count for creation and/or deletion of persistent
	burst buffers.
	-- Avoid terminating other processes in a task group when any task is core
	dumping to avoid incomplete OpenMP core files.
	-- Append ", with requeued tasks" to end of job array "end" email when any task
	is requeued. This is a hint to use "sacct --duplicates" to see all job
	accounting information.
	-- Consider nodes in "Resuming" state as available for backfill future
	scheduling and do not replace in advanced reservations.
	-- Explicitly shutdown the slurmd daemon when reboot requested.
	-- Calculate TRES billing values at job submission to enforce QOS DenyOnLimit
	configuration.
	-- Add acct_gather_profile/influxdb plugin to store job profiling information
	to InfluxDB rather than HDF5.