RELEASE_NOTES - SchedMD/slurm - Git at Google

 RELEASE NOTES FOR SLURM VERSION 14.03
 26 February 2014


 IMPORTANT NOTE:
 If using the slurmdbd (Slurm DataBase Daemon) you must update this first.
 The 14.03 slurmdbd will work with Slurm daemons of version 2.5 and above.
 You will not need to update all clusters at the same time, but it is very
 important to update slurmdbd first and having it running before updating
 any other clusters making use of it.  No real harm will come from updating
 your systems before the slurmdbd, but they will not talk to each other
 until you do.  Also at least the first time running the slurmdbd you need to
 make sure your my.cnf file has innodb_buffer_pool_size equal to at least 64M.
 You can accomplish this by adding the line

 innodb_buffer_pool_size=64M

 under the [mysqld] reference in the my.cnf file and restarting the mysqld.
 This is needed when converting large tables over to the new database schema.

 Slurm can be upgraded from version 2.5 or 2.6 to version 14.03 without loss of
 jobs or other state information. Upgrading directly from an earlier version of
 Slurm will result in loss of state information.


 HIGHLIGHTS
 ==========
  -- Added support for native Slurm operation on Cray systems (without ALPS).
  -- Added partition configuration parameters AllowAccounts, AllowQOS,
     DenyAccounts and DenyQOS to provide greater control over use.
  -- Added the ability to perform load based scheduling. Allocating resources to
     jobs on the nodes with the largest number if idle CPUs.
  -- Added support for reserving cores on a compute node for system services
     (core specialization)
  -- Add mechanism for job_submit plugin to generate error message for srun,
     salloc or sbatch to stderr.
  -- Support for Postgres database has long since been out of date and
     problematic, so it has been removed entirely.  If you would like to
     use it the code still exists in <= 2.6, but will not be included in
     this and future versions of the code.
  -- Added new structures and support for both server and cluster resources.
  -- Significant performance improvements, especially with respect to job
     array support.

 RPMBUILD CHANGES
 ================
  -- The rpmbuild option for a Cray system with ALPS has changed from
     %_with_cray to %_with_cray_alps.
  -- CRAY - Do not package Slurm's libpmi or libpmi2 libraries. The Cray version
     of those libraries must be used.


 CONFIGURATION FILE CHANGES (see man appropriate man page for details)
 =====================================================================
  -- Added AuthInfo configuration parameter to specify port used by
     authentication plugin.
  -- Added CoreSpecPlugin configuration parameter and core specialization plugin
     infrastructure.
  -- Added JobAcctGatherParams configuration parameter.
  -- Added JobContainerType configuration parameter and plugin infrastructure.
  -- Added FairShareDampeningFactor configuration parameter to offer a greater
     priority range based upon utilization.
  -- Added LogTimeFormat configuration parameter to control format of the
     timestamp in log files.
  -- Added PrologFlags configuration parameter to control when the Prolog is run.
  -- Added SlurmdPlugstack configuration parameter for generic slurmd daemon
     plugin mechanism.
  -- Added partition configuration parameters AllowAccounts, AllowQOS,
     DenyAccounts and DenyQOS to provide greater control over use.
  -- SelectType: select/cray has been renamed select/alps.
  -- Added SchedulingParameters paramter of "CR_LLN" and partition parameter of
     "LLN=yes|no" to enable loadbased scheduling.
  -- In SchedulerParameters, change default max_job_bf value from 50 to 100.
  -- Added SchedulerParameters option of "partition_job_depth" to limit
     scheduling logic depth by partition.
  -- Added SchedulerParameters option of "bf_max_job_start".
  -- In SchedulerParameters, replace "max_job_bf" with "bf_max_job_test"
     (both will work for now).
  -- Add SchedulerParameters options of "preempt_reorder_count" and
     "preempt_strict_order".
  -- Change MaxArraySize from 16-bit to 32-bit field.
  -- Added fields to "scontrol show job" output: boards_per_node,
     sockets_per_board, ntasks_per_node, ntasks_per_board, ntasks_per_socket,
     ntasks_per_core, and nice.
  -- gres.conf - Add "NodeName" specification so that a single gres.conf file
     can be used for a heterogeneous cluster.
  -- Added DebugFlags value of "License".

 DBD CONFIGURATION FILE CHANGES (see "man slurmdbd.conf" for details)
 ====================================================================


 COMMAND CHANGES (see man pages for details)
 ===========================================
  -- Added sbatch --signal option of "B:" to signal the batch shell rather than
     only the spawned job steps.
  -- Added sinfo and squeue format option of "%all" to print all fields available
     for the data type with a vertical bar separating each field.
  -- Add StdIn, StdOut, and StdErr paths to job information dumped with
     "scontrol show job".
  -- Permit Slurm administrator to submit a batch job as any user.
  -- Added -I|--item-extract option to sh5util to extract data item from series.
  -- Add squeue output format options for job command and working directory
     (%o and %Z respectively).
  -- Add stdin/out/err to sview job output.
  -- Added a new option to the scontrol command to view licenses that are configured
     in use and available. 'scontrol show licenses'.
  -- Permit jobs in finished state to be requeued.
  -- Added a new option to scontrol to put a requeued job on hold. A requeued
     job can be put in a new special state called SPECIAL_EXIT indicating
     the job has exited with a special value.
     "scontrol requeuehold state=SpecialExit 123".
  -- Pending job steps will have step_id of INFINITE rather than NO_VAL and
     will be reported as "TBD" by scontrol and squeue commands.
  -- Added sgather tool to gather files from a job's compute nodes into a
     central location.
  -- Added -S/--core-spec option to salloc, sbatch and srun commands to reserve
     specialized cores for system use.
  -- Added sacctmgr options to create, modify, list & delete Resources.

 OTHER CHANGES
 =============
  -- Added job_info() and step_info() functions to the gres plugins to extract
     plugin specific fields from the job's or step's GRES data structure.

 API CHANGES
 ===========
  -- Added "state" argument to slurm_requeue function.
  -- Added sacctmgr options to create, modify, list & delete Resources.

 Changed members of the following structs
 ========================================
  -- Struct job_info / slurm_job_info_t: Changed exc_node_inx, node_inx, and
     req_node_inx from type int to type int32_t; Changed array_task_id from type
     uint16_t to type uint32_t
  -- job_step_info_t: Changed node_inx from type int to type int32_t; Changed
     array_task_id from type uint16_t to type uint32_t; Changed node_inx from
     int to int32_t
  -- Struct partition_info / partition_info_t: Changed node_inx from type int
     to type int32_t
  -- block_job_info_t: Changed cnode_inx from type int to type int32_t
  -- block_info_t: Changed ionode_inx and mp_inx from type int to type int32_t
  -- Struct reserve_info / reserve_info_t: Changed node_inx from type int to
     type int32_t
  -- Struct slurm_ctl_conf / slurm_ctl_conf_t: Changed max_array_sz from uint16_t
     to uint32_t.
  -- Struct reserve_info / reserve_info_t: Changed flags from uint16_t to
     uint32_t.
  -- Struct resv_desc_msg / resv_desc_msg_t: Changed flags from uint16_t to
     uint32_t.

 Added the following struct definitions
 ======================================
  -- Struct job_info / slurm_job_info_t: Added std_err, std_in and std_out.
  -- Struct job_info / slurm_job_info_t: Added core_spec
  -- struct job_descriptor / job_desc_msg_t: Added core_spec, warn_flags
  -- Struct partition_info / partition_info_t: Added allow_accounts, allow_qos,
     deny_accounts, deny_groups and deny_qos.
  -- struct slurm_step_launch_params_t: Added partition (for Prolog environment
     variable)
  -- Struct resource_allocation_response_msg: Added partition (for Prolog
     environment variable)
  -- Struct slurm_ctl_conf / slurm_ctl_conf_t: Added acct_gather_conf, authinfo.
     core_spec_plugin, ext_sensors_conf, fs_dampening_factor,
     job_acct_gather_params, job_container_plugin, log_fmt, prolog_flags and
     slurmd_plugstack.
  -- Added license data structures: slurm_license_info and license_info_msg.

 Changed the following enums and #defines
 ========================================
  -- Add new job_state of JOB_BOOT_FAIL for job terminations due to failure to
     boot it's allocated nodes or BlueGene block.
  -- Added set of job state change flags: JOB_REQUEUE, JOB_REQUEUE_HOLD,
     JOB_SPECIAL_EXIT.
  -- Added new job_state_reason values: WAIT_BLOCK_MAX_ERR, WAIT_BLOCK_D_ACTION,
     WAIT_CLEANING and WAIT_PROLOG.
  -- Added new select_jobdata_type values: SELECT_JOBDATA_CLEANING and
     SELECT_JOBDATA_NETWORK.
  -- Added new node state values: NODE_STATE_NET, NODE_STATE_RES and
     NODE_STATE_UNDRAIN.
  -- Added new flags for SelectTypeParameters: CR_OTHER_CONS_RES, CR_NHC_STEP_NO,
     CR_LLN, and CR_NHC_NO.
  -- Added new PriorityFlags value of PRIORITY_FLAGS_DEPTH_OBLIVIOUS
  -- Added new partition flags: PART_FLAG_LLN and PART_FLAG_LLN_CLR
  -- Added DebugFlags: DEBUG_FLAG_JOB_CONT and DEBUG_FLAG_TASK
  -- Removed DebugFlags: DEBUG_FLAG_THREADID
  -- Added PrologFlag: PROLOG_FLAG_ALLOC
  -- Added LogTimeFormat options: LOG_FMT_ISO8601_MS, LOG_FMT_ISO8601,
     LOG_FMT_RFC5424_MS, LOG_FMT_RFC5424, LOG_FMT_CLOCK, LOG_FMT_SHORT and
     LOG_FMT_THREAD_ID

 Added the following API's
 =========================
  -- Added functions for retrieving license information: slurm_load_licenses and
     slurm_free_license_info_msg.
  -- Added functions to get specific job information: slurm_get_job_stderr,
     slurm_get_job_stdin and slurm_get_job_stdout.

 Change the following API's
 ===========================
  -- Add task pointer to the task_post_term() function in task plugins. The
     terminating task's PID is available in task->pid.


 DBD API Changes
 ===============

 Changed members of the following structs
 ========================================

 Added the following struct definitions
 ======================================
  -- struct slurmdb_clus_res_rec_t: new cluster resource record.
  -- struct slurmdb_res_cond_t : new resource condition.
  -- struct slurmdb_res_rec_t: new resource record.

 Added the following enums and #defines
 ========================================
  -- enum slurmdb_resource_type_t: server resource record "type" definition.
  -- Added new slurmdb_update_type_t SLURMDB_ADD_RES,
     SLURMDB_REMOVE_RES, SLURMDB_MODIFY_RES, SLURMDB_REMOVE_QOS_USAGE
  -- Added new flags for Resources SLURMDB_RES_FLAG_BASE,
     SLURMDB_RES_FLAG_NOTSET, SLURMDB_RES_FLAG_ADD, SLURMDB_RES_FLAG_REMOVE
  -- Added new flags for QOS QOS_FLAG_BASE

 Added the following API's
 =========================
  -- Added functions for resource management in the database.
	RELEASE NOTES FOR SLURM VERSION 14.03
	26 February 2014


	IMPORTANT NOTE:
	If using the slurmdbd (Slurm DataBase Daemon) you must update this first.
	The 14.03 slurmdbd will work with Slurm daemons of version 2.5 and above.
	You will not need to update all clusters at the same time, but it is very
	important to update slurmdbd first and having it running before updating
	any other clusters making use of it. No real harm will come from updating
	your systems before the slurmdbd, but they will not talk to each other
	until you do. Also at least the first time running the slurmdbd you need to
	make sure your my.cnf file has innodb_buffer_pool_size equal to at least 64M.
	You can accomplish this by adding the line

	innodb_buffer_pool_size=64M

	under the [mysqld] reference in the my.cnf file and restarting the mysqld.
	This is needed when converting large tables over to the new database schema.

	Slurm can be upgraded from version 2.5 or 2.6 to version 14.03 without loss of
	jobs or other state information. Upgrading directly from an earlier version of
	Slurm will result in loss of state information.


	HIGHLIGHTS
	==========
	-- Added support for native Slurm operation on Cray systems (without ALPS).
	-- Added partition configuration parameters AllowAccounts, AllowQOS,
	DenyAccounts and DenyQOS to provide greater control over use.
	-- Added the ability to perform load based scheduling. Allocating resources to
	jobs on the nodes with the largest number if idle CPUs.
	-- Added support for reserving cores on a compute node for system services
	(core specialization)
	-- Add mechanism for job_submit plugin to generate error message for srun,
	salloc or sbatch to stderr.
	-- Support for Postgres database has long since been out of date and
	problematic, so it has been removed entirely. If you would like to
	use it the code still exists in <= 2.6, but will not be included in
	this and future versions of the code.
	-- Added new structures and support for both server and cluster resources.
	-- Significant performance improvements, especially with respect to job
	array support.

	RPMBUILD CHANGES
	================
	-- The rpmbuild option for a Cray system with ALPS has changed from
	%_with_cray to %_with_cray_alps.
	-- CRAY - Do not package Slurm's libpmi or libpmi2 libraries. The Cray version
	of those libraries must be used.


	CONFIGURATION FILE CHANGES (see man appropriate man page for details)
	=====================================================================
	-- Added AuthInfo configuration parameter to specify port used by
	authentication plugin.
	-- Added CoreSpecPlugin configuration parameter and core specialization plugin
	infrastructure.
	-- Added JobAcctGatherParams configuration parameter.
	-- Added JobContainerType configuration parameter and plugin infrastructure.
	-- Added FairShareDampeningFactor configuration parameter to offer a greater
	priority range based upon utilization.
	-- Added LogTimeFormat configuration parameter to control format of the
	timestamp in log files.
	-- Added PrologFlags configuration parameter to control when the Prolog is run.
	-- Added SlurmdPlugstack configuration parameter for generic slurmd daemon
	plugin mechanism.
	-- Added partition configuration parameters AllowAccounts, AllowQOS,
	DenyAccounts and DenyQOS to provide greater control over use.
	-- SelectType: select/cray has been renamed select/alps.
	-- Added SchedulingParameters paramter of "CR_LLN" and partition parameter of
	"LLN=yes\|no" to enable loadbased scheduling.
	-- In SchedulerParameters, change default max_job_bf value from 50 to 100.
	-- Added SchedulerParameters option of "partition_job_depth" to limit
	scheduling logic depth by partition.
	-- Added SchedulerParameters option of "bf_max_job_start".
	-- In SchedulerParameters, replace "max_job_bf" with "bf_max_job_test"
	(both will work for now).
	-- Add SchedulerParameters options of "preempt_reorder_count" and
	"preempt_strict_order".
	-- Change MaxArraySize from 16-bit to 32-bit field.
	-- Added fields to "scontrol show job" output: boards_per_node,
	sockets_per_board, ntasks_per_node, ntasks_per_board, ntasks_per_socket,
	ntasks_per_core, and nice.
	-- gres.conf - Add "NodeName" specification so that a single gres.conf file
	can be used for a heterogeneous cluster.
	-- Added DebugFlags value of "License".

	DBD CONFIGURATION FILE CHANGES (see "man slurmdbd.conf" for details)
	====================================================================


	COMMAND CHANGES (see man pages for details)
	===========================================
	-- Added sbatch --signal option of "B:" to signal the batch shell rather than
	only the spawned job steps.
	-- Added sinfo and squeue format option of "%all" to print all fields available
	for the data type with a vertical bar separating each field.
	-- Add StdIn, StdOut, and StdErr paths to job information dumped with
	"scontrol show job".
	-- Permit Slurm administrator to submit a batch job as any user.
	-- Added -I\|--item-extract option to sh5util to extract data item from series.
	-- Add squeue output format options for job command and working directory
	(%o and %Z respectively).
	-- Add stdin/out/err to sview job output.
	-- Added a new option to the scontrol command to view licenses that are configured
	in use and available. 'scontrol show licenses'.
	-- Permit jobs in finished state to be requeued.
	-- Added a new option to scontrol to put a requeued job on hold. A requeued
	job can be put in a new special state called SPECIAL_EXIT indicating
	the job has exited with a special value.
	"scontrol requeuehold state=SpecialExit 123".
	-- Pending job steps will have step_id of INFINITE rather than NO_VAL and
	will be reported as "TBD" by scontrol and squeue commands.
	-- Added sgather tool to gather files from a job's compute nodes into a
	central location.
	-- Added -S/--core-spec option to salloc, sbatch and srun commands to reserve
	specialized cores for system use.
	-- Added sacctmgr options to create, modify, list & delete Resources.

	OTHER CHANGES
	=============
	-- Added job_info() and step_info() functions to the gres plugins to extract
	plugin specific fields from the job's or step's GRES data structure.

	API CHANGES
	===========
	-- Added "state" argument to slurm_requeue function.
	-- Added sacctmgr options to create, modify, list & delete Resources.

	Changed members of the following structs
	========================================
	-- Struct job_info / slurm_job_info_t: Changed exc_node_inx, node_inx, and
	req_node_inx from type int to type int32_t; Changed array_task_id from type
	uint16_t to type uint32_t
	-- job_step_info_t: Changed node_inx from type int to type int32_t; Changed
	array_task_id from type uint16_t to type uint32_t; Changed node_inx from
	int to int32_t
	-- Struct partition_info / partition_info_t: Changed node_inx from type int
	to type int32_t
	-- block_job_info_t: Changed cnode_inx from type int to type int32_t
	-- block_info_t: Changed ionode_inx and mp_inx from type int to type int32_t
	-- Struct reserve_info / reserve_info_t: Changed node_inx from type int to
	type int32_t
	-- Struct slurm_ctl_conf / slurm_ctl_conf_t: Changed max_array_sz from uint16_t
	to uint32_t.
	-- Struct reserve_info / reserve_info_t: Changed flags from uint16_t to
	uint32_t.
	-- Struct resv_desc_msg / resv_desc_msg_t: Changed flags from uint16_t to
	uint32_t.

	Added the following struct definitions
	======================================
	-- Struct job_info / slurm_job_info_t: Added std_err, std_in and std_out.
	-- Struct job_info / slurm_job_info_t: Added core_spec
	-- struct job_descriptor / job_desc_msg_t: Added core_spec, warn_flags
	-- Struct partition_info / partition_info_t: Added allow_accounts, allow_qos,
	deny_accounts, deny_groups and deny_qos.
	-- struct slurm_step_launch_params_t: Added partition (for Prolog environment
	variable)
	-- Struct resource_allocation_response_msg: Added partition (for Prolog
	environment variable)
	-- Struct slurm_ctl_conf / slurm_ctl_conf_t: Added acct_gather_conf, authinfo.
	core_spec_plugin, ext_sensors_conf, fs_dampening_factor,
	job_acct_gather_params, job_container_plugin, log_fmt, prolog_flags and
	slurmd_plugstack.
	-- Added license data structures: slurm_license_info and license_info_msg.

	Changed the following enums and #defines
	========================================
	-- Add new job_state of JOB_BOOT_FAIL for job terminations due to failure to
	boot it's allocated nodes or BlueGene block.
	-- Added set of job state change flags: JOB_REQUEUE, JOB_REQUEUE_HOLD,
	JOB_SPECIAL_EXIT.
	-- Added new job_state_reason values: WAIT_BLOCK_MAX_ERR, WAIT_BLOCK_D_ACTION,
	WAIT_CLEANING and WAIT_PROLOG.
	-- Added new select_jobdata_type values: SELECT_JOBDATA_CLEANING and
	SELECT_JOBDATA_NETWORK.
	-- Added new node state values: NODE_STATE_NET, NODE_STATE_RES and
	NODE_STATE_UNDRAIN.
	-- Added new flags for SelectTypeParameters: CR_OTHER_CONS_RES, CR_NHC_STEP_NO,
	CR_LLN, and CR_NHC_NO.
	-- Added new PriorityFlags value of PRIORITY_FLAGS_DEPTH_OBLIVIOUS
	-- Added new partition flags: PART_FLAG_LLN and PART_FLAG_LLN_CLR
	-- Added DebugFlags: DEBUG_FLAG_JOB_CONT and DEBUG_FLAG_TASK
	-- Removed DebugFlags: DEBUG_FLAG_THREADID
	-- Added PrologFlag: PROLOG_FLAG_ALLOC
	-- Added LogTimeFormat options: LOG_FMT_ISO8601_MS, LOG_FMT_ISO8601,
	LOG_FMT_RFC5424_MS, LOG_FMT_RFC5424, LOG_FMT_CLOCK, LOG_FMT_SHORT and
	LOG_FMT_THREAD_ID

	Added the following API's
	=========================
	-- Added functions for retrieving license information: slurm_load_licenses and
	slurm_free_license_info_msg.
	-- Added functions to get specific job information: slurm_get_job_stderr,
	slurm_get_job_stdin and slurm_get_job_stdout.

	Change the following API's
	===========================
	-- Add task pointer to the task_post_term() function in task plugins. The
	terminating task's PID is available in task->pid.



	DBD API Changes
	===============

	Changed members of the following structs
	========================================

	Added the following struct definitions
	======================================
	-- struct slurmdb_clus_res_rec_t: new cluster resource record.
	-- struct slurmdb_res_cond_t : new resource condition.
	-- struct slurmdb_res_rec_t: new resource record.

	Added the following enums and #defines
	========================================
	-- enum slurmdb_resource_type_t: server resource record "type" definition.
	-- Added new slurmdb_update_type_t SLURMDB_ADD_RES,
	SLURMDB_REMOVE_RES, SLURMDB_MODIFY_RES, SLURMDB_REMOVE_QOS_USAGE
	-- Added new flags for Resources SLURMDB_RES_FLAG_BASE,
	SLURMDB_RES_FLAG_NOTSET, SLURMDB_RES_FLAG_ADD, SLURMDB_RES_FLAG_REMOVE
	-- Added new flags for QOS QOS_FLAG_BASE

	Added the following API's
	=========================
	-- Added functions for resource management in the database.