testsuite/expect/README - SchedMD/slurm - Git at Google

 ############################################################################
 # Copyright (C) 2002-2007 The Regents of the University of California.
 # Copyright (C) 2008-2011 Lawrence Livermore National Security.
 # Copyright (C) SchedMD LLC.
 # Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
 # Written by Morris Jette <jette1@llnl.gov>
 # Additions by Joseph Donaghy <donaghy1@llnl.gov>
 # CODE-OCEC-09-009. All rights reserved.
 #
 # This file is part of Slurm, a resource management program.
 # For details, see <https://slurm.schedmd.com/>.
 # Please also read the supplied file: DISCLAIMER.
 #
 # Slurm is free software; you can redistribute it and/or modify it under
 # the terms of the GNU General Public License as published by the Free
 # Software Foundation; either version 2 of the License, or (at your option)
 # any later version.
 #
 # Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
 # FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
 # details.
 #
 # You should have received a copy of the GNU General Public License along
 # with Slurm; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301  USA.
 ############################################################################

 This directory contains a battery of Slurm regression tests. The tests make
 use of the "expect" scripting language.

 The expect tests first look for the (unified) testsuite.conf configuration
 file. By default, this is expected to be found in SRCDIR/testsuite. If placed
 elsewhere, the SLURM_TESTSUITE_CONF environment variable should be set to the
 full path of the testsuite.conf file. This file can be created from a copy of
 the autogenerated sample found in BUILDDIR/testsuite/testsuite.conf.sample
 and then edited as necessary.

 The expect tests next look for an (expect-specific) globals.local configuration
 file. Values in this file override values in the testsuite.conf file. The
 globals.local file can be in the directory containing this README, or
 the one configured in the environment variable SLURM_LOCAL_GLOBALS_FILE.

 The globals.local file contains (expect) testsuite parameters, for example:

 slurm_dir:                    slurm installation directory (prefix) [/usr]
 build_dir:                    slurm build directory [../..]
 testsuite_cleanup_on_failure: run cleanup and remove test_dir on failure [true]
 testsuite_shared_dir:         cluster-wide shared directory [.]
 testsuite_log_level:          logging threshold [$LOG_LEVEL_DEBUG]
 testsuite_log_format:         specifies the tracing fields used in logging
 testsuite_colorize:           boolean that turns colorization on or off
 testsuite_subtest_fatal:      first failing subtest aborts the test [false]
 testsuite_subtest_details:    add details of subtest in the summary [fail]
                               possible values: all, fail, fail_skip or none
 testsuite_testproc_details:   add details of testprocs in the summary [fail]
                               possible values: all, fail, fail_skip or none
 testsuite_testproc_log_calls: yes to always print the whole testproc call,
                               no to never print it, or [ignore_skips] to print
                               all but the skipped testprocs
 testsuite_user:               a system username that is NOT on the Slurm DB
                               that may be used to sudo commands as him
                               instead of current user (see check_run_as_user)

 The testsuite needs at least one cluster-wide shared file system.
 By default, we assume that it's the testsuite/expect directory (i.e. ".").
 Files created within this directory on one node in the cluster should be
 visible to all other nodes in the cluster.

 For the valid fields usable with testsuite_log_format, see the header
 comments for the _log_format procedure. See also the default definition of
 testsuite_log_format in globals for an example of syntax.

 Each test can be executed independently. Upon successful completion, the test
 will print "SUCCESS" and terminate with an exit code of zero. Upon failure,
 the test will print "FAILURE" and terminate with a positive exit code. If a
 test fails its precondition checks, it will print "SKIPPED" and terminate with
 a negative exit code. Use the fail, skip or pass procedures to end the test.

 A test can expect to have some global variables defined that it can make use
 of within the test. These include:

 test_name:  the name of the test script, e.g. test1.1
 test_id:    the numeric suffix for the test, e.g. 1.1
 test_dir:   a temporary subdirectory created within the testsuite_shared_dir *

 * The test_dir will get removed at the end of the test unless the test failed
   and testsuite_cleanup_on_failure was set to false. The test_dir will be
   (removed and) recreated the next time the test is run.

 A test can be run as normal executable (uses expect) and can read by default
 two main arguments:
 	-i : a comma-separated list of testproc ids to run
 	-e : a comma-separated list of testproc ids to skip
 These arguments will be removed and not available on argv on the tests.
 Test writers should use testproc to launch their testproc functions to
 take advantage of those arguments (see testproc function).

 The script "regression.py" will execute all of the tests and summarize the
 results. Standard output contains detailed logging of all events, which is
 quite verbose. Failure information is written to standard error. A good
 way to run "regression.py" is to write its standard output to one file and
 either write standard error to another file or print it to the terminal.
 Execution time of the full test suite is roughly 80 minutes, but can vary
 considerably with the architecture, configuration, and system load. Some
 tests send e-mail, so check for four e-mail messages sent to the user
 running the tests.

 When failures do occur, check the standard output for details. Searching
 for the keyword "FAILURE" will typically locate the failing test. Note
 that some of the tests are architecture or configuration specific.  Also
 note that most tests are designed to be run as a normal user. Tests 3.#
 are designed to be run as user root or SlurmUser, but will be skipped
 when the full test suite is executed as an unprivileged user. The full
 test suite is typically executed many times by the Slurm developers on a
 variety of systems before a Slurm release is made. This has resulted in
 high system reliability. When Slurm bugs are found or features added,
 this test suite is expanded.
	############################################################################
	# Copyright (C) 2002-2007 The Regents of the University of California.
	# Copyright (C) 2008-2011 Lawrence Livermore National Security.
	# Copyright (C) SchedMD LLC.
	# Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	# Written by Morris Jette <jette1@llnl.gov>
	# Additions by Joseph Donaghy <donaghy1@llnl.gov>
	# CODE-OCEC-09-009. All rights reserved.
	#
	# This file is part of Slurm, a resource management program.
	# For details, see <https://slurm.schedmd.com/>.
	# Please also read the supplied file: DISCLAIMER.
	#
	# Slurm is free software; you can redistribute it and/or modify it under
	# the terms of the GNU General Public License as published by the Free
	# Software Foundation; either version 2 of the License, or (at your option)
	# any later version.
	#
	# Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
	# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
	# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
	# details.
	#
	# You should have received a copy of the GNU General Public License along
	# with Slurm; if not, write to the Free Software Foundation, Inc.,
	# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
	############################################################################

	This directory contains a battery of Slurm regression tests. The tests make
	use of the "expect" scripting language.

	The expect tests first look for the (unified) testsuite.conf configuration
	file. By default, this is expected to be found in SRCDIR/testsuite. If placed
	elsewhere, the SLURM_TESTSUITE_CONF environment variable should be set to the
	full path of the testsuite.conf file. This file can be created from a copy of
	the autogenerated sample found in BUILDDIR/testsuite/testsuite.conf.sample
	and then edited as necessary.

	The expect tests next look for an (expect-specific) globals.local configuration
	file. Values in this file override values in the testsuite.conf file. The
	globals.local file can be in the directory containing this README, or
	the one configured in the environment variable SLURM_LOCAL_GLOBALS_FILE.

	The globals.local file contains (expect) testsuite parameters, for example:

	slurm_dir: slurm installation directory (prefix) [/usr]
	build_dir: slurm build directory [../..]
	testsuite_cleanup_on_failure: run cleanup and remove test_dir on failure [true]
	testsuite_shared_dir: cluster-wide shared directory [.]
	testsuite_log_level: logging threshold [$LOG_LEVEL_DEBUG]
	testsuite_log_format: specifies the tracing fields used in logging
	testsuite_colorize: boolean that turns colorization on or off
	testsuite_subtest_fatal: first failing subtest aborts the test [false]
	testsuite_subtest_details: add details of subtest in the summary [fail]
	possible values: all, fail, fail_skip or none
	testsuite_testproc_details: add details of testprocs in the summary [fail]
	possible values: all, fail, fail_skip or none
	testsuite_testproc_log_calls: yes to always print the whole testproc call,
	no to never print it, or [ignore_skips] to print
	all but the skipped testprocs
	testsuite_user: a system username that is NOT on the Slurm DB
	that may be used to sudo commands as him
	instead of current user (see check_run_as_user)

	The testsuite needs at least one cluster-wide shared file system.
	By default, we assume that it's the testsuite/expect directory (i.e. ".").
	Files created within this directory on one node in the cluster should be
	visible to all other nodes in the cluster.

	For the valid fields usable with testsuite_log_format, see the header
	comments for the _log_format procedure. See also the default definition of
	testsuite_log_format in globals for an example of syntax.

	Each test can be executed independently. Upon successful completion, the test
	will print "SUCCESS" and terminate with an exit code of zero. Upon failure,
	the test will print "FAILURE" and terminate with a positive exit code. If a
	test fails its precondition checks, it will print "SKIPPED" and terminate with
	a negative exit code. Use the fail, skip or pass procedures to end the test.

	A test can expect to have some global variables defined that it can make use
	of within the test. These include:

	test_name: the name of the test script, e.g. test1.1
	test_id: the numeric suffix for the test, e.g. 1.1
	test_dir: a temporary subdirectory created within the testsuite_shared_dir *

	* The test_dir will get removed at the end of the test unless the test failed
	and testsuite_cleanup_on_failure was set to false. The test_dir will be
	(removed and) recreated the next time the test is run.

	A test can be run as normal executable (uses expect) and can read by default
	two main arguments:
	-i : a comma-separated list of testproc ids to run
	-e : a comma-separated list of testproc ids to skip
	These arguments will be removed and not available on argv on the tests.
	Test writers should use testproc to launch their testproc functions to
	take advantage of those arguments (see testproc function).

	The script "regression.py" will execute all of the tests and summarize the
	results. Standard output contains detailed logging of all events, which is
	quite verbose. Failure information is written to standard error. A good
	way to run "regression.py" is to write its standard output to one file and
	either write standard error to another file or print it to the terminal.
	Execution time of the full test suite is roughly 80 minutes, but can vary
	considerably with the architecture, configuration, and system load. Some
	tests send e-mail, so check for four e-mail messages sent to the user
	running the tests.

	When failures do occur, check the standard output for details. Searching
	for the keyword "FAILURE" will typically locate the failing test. Note
	that some of the tests are architecture or configuration specific. Also
	note that most tests are designed to be run as a normal user. Tests 3.#
	are designed to be run as user root or SlurmUser, but will be skipped
	when the full test suite is executed as an unprivileged user. The full
	test suite is typically executed many times by the Slurm developers on a
	variety of systems before a Slurm release is made. This has resulted in
	high system reliability. When Slurm bugs are found or features added,
	this test suite is expanded.