zfs/tests/test-runner/man/test-runner.1 - backupdr - Git at Google

 .\"
 .\" This file and its contents are supplied under the terms of the
 .\" Common Development and Distribution License ("CDDL"), version 1.0.
 .\" You may only use this file in accordance with the terms of version
 .\" 1.0 of the CDDL.
 .\"
 .\" A full copy of the text of the CDDL should have accompanied this
 .\" source.  A copy of the CDDL is also available via the Internet at
 .\" http://www.illumos.org/license/CDDL.
 .\"
 .\" Copyright (c) 2012 by Delphix. All rights reserved.
 .\"
 .Dd May 26, 2021
 .Dt RUN 1
 .Os
 .
 .Sh NAME
 .Nm run
 .Nd find, execute, and log the results of tests
 .Sh SYNOPSIS
 .Nm
 .Op Fl dgq
 .Op Fl o Ar outputdir
 .Op Fl pP Ar script
 .Op Fl t seconds
 .Op Fl uxX Ar username
 .Ar pathname Ns No …
 .Pp
 .Nm
 .Fl w Ar runfile
 .Op Fl gq
 .Op Fl o Ar outputdir
 .Op Fl pP Ar script
 .Op Fl t seconds
 .Op Fl uxX Ar username
 .Ar pathname Ns No …
 .Pp
 .Nm
 .Fl c Ar runfile
 .Op Fl dq
 .Pp
 .Nm
 .Op Fl h
 .
 .Sh DESCRIPTION
 .Nm
 command has three basic modes of operation.
 With neither
 .Fl c
 nor
 .Fl w ,
 .Nm
 processes the arguments provided on
 the command line, adding them to the list for this run.
 If a specified
 .Ar pathname
 is an executable file, it is added as a test.
 If a specified
 .Ar pathname
 is a directory, the behavior depends upon the presence of
 .Fl g .
 If
 .Fl g
 is specified, the directory is treated as a test group.
 See the section on
 .Sy Test Groups
 below.
 Without
 .Fl g ,
 .Nm
 simply descends into the directory looking for executable files.
 The tests are then executed, and the results are logged.
 .Pp
 With
 .Fl w ,
 .Nm
 finds tests in the manner described above.
 Rather than executing the tests and logging the results, the test configuration
 is stored in a
 .Ar runfile ,
 which can be used in future invocations, or edited
 to modify which tests are executed and which options are applied.
 Options included on the command line with
 .Fl w
 become defaults in the
 .Ar runfile .
 .Pp
 With
 .Fl c ,
 .Nm
 parses a
 .Ar runfile ,
 which can specify a series of tests and test groups to be executed.
 The tests are then executed, and the results are logged.
 .
 .Ss Test Groups
 A test group is comprised of a set of executable files, all of which exist in
 one directory.
 The options specified on the command line or in a
 .Ar runfile
 apply to individual tests in the group.
 The exception is options pertaining to pre and post scripts, which act on all tests as a group.
 Rather than running before and after each test,
 these scripts are run only once each at the start and end of the test group.
 .Ss Test Execution
 The specified tests run serially, and are typically assigned results according
 to exit values.
 Tests that exit zero and non-zero are marked
 .Sy PASS
 and
 .Sy FAIL ,
 respectively.
 When a pre script fails for a test group, only the post script is executed,
 and the remaining tests are marked
 .Sy SKIPPED .
 Any test that exceeds
 its
 .Ar timeout
 is terminated, and marked
 .Sy KILLED .
 .Pp
 By default, tests are executed with the credentials of the
 .Nm
 script.
 Executing tests with other credentials is done via
 .Xr sudo 1m ,
 which must
 be configured to allow execution without prompting for a password.
 Environment variables from the calling shell are available to individual tests.
 During test execution, the working directory is changed to
 .Ar outputdir .
 .
 .Ss Output Logging
 By default,
 .Nm
 will print one line on standard output at the conclusion
 of each test indicating the test name, result and elapsed time.
 Additionally, for each invocation of
 .Nm ,
 a directory is created using the ISO 8601 date format.
 Within this directory is a file named
 .Sy log
 containing all the
 test output with timestamps, and a directory for each test.
 Within the test directories, there is one file each for standard output,
 standard error and merged output.
 The default location for the
 .Ar outputdir
 is
 .Pa /var/tmp/test_results .
 .Ss "Runfiles"
 The
 .Ar runfile
 is an INI-style configuration file that describes a test run.
 The file has one section named
 .Sy DEFAULT ,
 which contains configuration option
 names and their values in
 .Sy name No = Ar value
 format.
 The values in this section apply to all the subsequent sections,
 unless they are also specified there, in which case the default is overridden.
 The remaining section names are the absolute pathnames of files and directories,
 describing tests and test groups respectively.
 The legal option names are:
 .Bl -tag -width "tests = ['filename', …]"
 .It Sy outputdir No = Ar pathname
 The name of the directory that holds test logs.
 .It Sy pre No = Ar script
 Run
 .Ar script
 prior to the test or test group.
 .It Sy pre_user No = Ar username
 Execute the pre script as
 .Ar username .
 .It Sy post No = Ar script
 Run
 .Ar script
 after the test or test group.
 .It Sy post_user No = Ar username
 Execute the post script as
 .Ar username .
 .It Sy quiet No = Sy True Ns | Ns Sy False
 If
 .Sy True ,
 only the results summary is printed to standard out.
 .It Sy tests No = [ Ns Ar 'filename' , No … ]
 Specify a list of
 .Ar filenames
 for this test group.
 Only the basename of the absolute path is required.
 This option is only valid for test groups, and each
 .Ar filename
 must be single quoted.
 .It Sy timeout No = Ar n
 A timeout value of
 .Ar n
 seconds.
 .It Sy user No = Ar username
 Execute the test or test group as
 .Ar username .
 .El
 .
 .Sh OPTIONS
 .Bl -tag -width "-o outputdir"
 .It Fl c Ar runfile
 Specify a
 .Ar runfile
 to be consumed by the run command.
 .It Fl d
 Dry run mode.
 Execute no tests, but print a description of each test that would have been run.
 .It Fl m
 Enable kmemleak reporting (Linux only)
 .It Fl g
 Create test groups from any directories found while searching for tests.
 .It Fl o Ar outputdir
 Specify the directory in which to write test results.
 .It Fl p Ar script
 Run
 .Ar script
 prior to any test or test group.
 .It Fl P Ar script
 Run
 .Ar script
 after any test or test group.
 .It Fl q
 Print only the results summary to the standard output.
 .It Fl s Ar script
 Run
 .Ar script
 as a failsafe after any test is killed.
 .It Fl S Ar username
 Execute the failsafe script as
 .Ar username .
 .It Fl t Ar n
 Specify a timeout value of
 .Ar n
 seconds per test.
 .It Fl u Ar username
 Execute tests or test groups as
 .Ar username .
 .It Fl w Ar runfile
 Specify the name of the
 .Ar runfile
 to create.
 .It Fl x Ar username
 Execute the pre script as
 .Ar username .
 .It Fl X Ar username
 Execute the post script as
 .Ar username .
 .El
 .
 .Sh EXAMPLES
 .Bl -tag -width "-h"
 .It Sy Example 1 : No Running ad-hoc tests.
 This example demonstrates the simplest invocation of
 .Nm .
 .Bd -literal
 .No % Nm run Ar my-tests
 Test: /home/jkennedy/my-tests/test-01                    [00:02] [PASS]
 Test: /home/jkennedy/my-tests/test-02                    [00:04] [PASS]
 Test: /home/jkennedy/my-tests/test-03                    [00:01] [PASS]

 Results Summary
 PASS       3

 Running Time:   00:00:07
 Percent passed: 100.0%
 Log directory:  /var/tmp/test_results/20120923T180654
 .Ed
 .It Sy Example 2 : No Creating a Ar runfile No for future use.
 This example demonstrates creating a
 .Ar runfile
 with non-default options.
 .Bd -literal
 .No % Nm run Fl p Ar setup Fl x Ar root Fl g Fl w Ar new-tests.run Ar new-tests
 .No % Nm cat Pa new-tests.run
 [DEFAULT]
 pre = setup
 post_user =
 quiet = False
 user =
 timeout = 60
 post =
 pre_user = root
 outputdir = /var/tmp/test_results

 [/home/jkennedy/new-tests]
 tests = ['test-01', 'test-02', 'test-03']
 .Ed
 .El
 .
 .Sh SEE ALSO
 .Xr sudo 1m
	.\"
	.\" This file and its contents are supplied under the terms of the
	.\" Common Development and Distribution License ("CDDL"), version 1.0.
	.\" You may only use this file in accordance with the terms of version
	.\" 1.0 of the CDDL.
	.\"
	.\" A full copy of the text of the CDDL should have accompanied this
	.\" source. A copy of the CDDL is also available via the Internet at
	.\" http://www.illumos.org/license/CDDL.
	.\"
	.\" Copyright (c) 2012 by Delphix. All rights reserved.
	.\"
	.Dd May 26, 2021
	.Dt RUN 1
	.Os
	.
	.Sh NAME
	.Nm run
	.Nd find, execute, and log the results of tests
	.Sh SYNOPSIS
	.Nm
	.Op Fl dgq
	.Op Fl o Ar outputdir
	.Op Fl pP Ar script
	.Op Fl t seconds
	.Op Fl uxX Ar username
	.Ar pathname Ns No …
	.Pp
	.Nm
	.Fl w Ar runfile
	.Op Fl gq
	.Op Fl o Ar outputdir
	.Op Fl pP Ar script
	.Op Fl t seconds
	.Op Fl uxX Ar username
	.Ar pathname Ns No …
	.Pp
	.Nm
	.Fl c Ar runfile
	.Op Fl dq
	.Pp
	.Nm
	.Op Fl h
	.
	.Sh DESCRIPTION
	.Nm
	command has three basic modes of operation.
	With neither
	.Fl c
	nor
	.Fl w ,
	.Nm
	processes the arguments provided on
	the command line, adding them to the list for this run.
	If a specified
	.Ar pathname
	is an executable file, it is added as a test.
	If a specified
	.Ar pathname
	is a directory, the behavior depends upon the presence of
	.Fl g .
	If
	.Fl g
	is specified, the directory is treated as a test group.
	See the section on
	.Sy Test Groups
	below.
	Without
	.Fl g ,
	.Nm
	simply descends into the directory looking for executable files.
	The tests are then executed, and the results are logged.
	.Pp
	With
	.Fl w ,
	.Nm
	finds tests in the manner described above.
	Rather than executing the tests and logging the results, the test configuration
	is stored in a
	.Ar runfile ,
	which can be used in future invocations, or edited
	to modify which tests are executed and which options are applied.
	Options included on the command line with
	.Fl w
	become defaults in the
	.Ar runfile .
	.Pp
	With
	.Fl c ,
	.Nm
	parses a
	.Ar runfile ,
	which can specify a series of tests and test groups to be executed.
	The tests are then executed, and the results are logged.
	.
	.Ss Test Groups
	A test group is comprised of a set of executable files, all of which exist in
	one directory.
	The options specified on the command line or in a
	.Ar runfile
	apply to individual tests in the group.
	The exception is options pertaining to pre and post scripts, which act on all tests as a group.
	Rather than running before and after each test,
	these scripts are run only once each at the start and end of the test group.
	.Ss Test Execution
	The specified tests run serially, and are typically assigned results according
	to exit values.
	Tests that exit zero and non-zero are marked
	.Sy PASS
	and
	.Sy FAIL ,
	respectively.
	When a pre script fails for a test group, only the post script is executed,
	and the remaining tests are marked
	.Sy SKIPPED .
	Any test that exceeds
	its
	.Ar timeout
	is terminated, and marked
	.Sy KILLED .
	.Pp
	By default, tests are executed with the credentials of the
	.Nm
	script.
	Executing tests with other credentials is done via
	.Xr sudo 1m ,
	which must
	be configured to allow execution without prompting for a password.
	Environment variables from the calling shell are available to individual tests.
	During test execution, the working directory is changed to
	.Ar outputdir .
	.
	.Ss Output Logging
	By default,
	.Nm
	will print one line on standard output at the conclusion
	of each test indicating the test name, result and elapsed time.
	Additionally, for each invocation of
	.Nm ,
	a directory is created using the ISO 8601 date format.
	Within this directory is a file named
	.Sy log
	containing all the
	test output with timestamps, and a directory for each test.
	Within the test directories, there is one file each for standard output,
	standard error and merged output.
	The default location for the
	.Ar outputdir
	is
	.Pa /var/tmp/test_results .
	.Ss "Runfiles"
	The
	.Ar runfile
	is an INI-style configuration file that describes a test run.
	The file has one section named
	.Sy DEFAULT ,
	which contains configuration option
	names and their values in
	.Sy name No = Ar value
	format.
	The values in this section apply to all the subsequent sections,
	unless they are also specified there, in which case the default is overridden.
	The remaining section names are the absolute pathnames of files and directories,
	describing tests and test groups respectively.
	The legal option names are:
	.Bl -tag -width "tests = ['filename', …]"
	.It Sy outputdir No = Ar pathname
	The name of the directory that holds test logs.
	.It Sy pre No = Ar script
	Run
	.Ar script
	prior to the test or test group.
	.It Sy pre_user No = Ar username
	Execute the pre script as
	.Ar username .
	.It Sy post No = Ar script
	Run
	.Ar script
	after the test or test group.
	.It Sy post_user No = Ar username
	Execute the post script as
	.Ar username .
	.It Sy quiet No = Sy True Ns \| Ns Sy False
	If
	.Sy True ,
	only the results summary is printed to standard out.
	.It Sy tests No = [ Ns Ar 'filename' , No … ]
	Specify a list of
	.Ar filenames
	for this test group.
	Only the basename of the absolute path is required.
	This option is only valid for test groups, and each
	.Ar filename
	must be single quoted.
	.It Sy timeout No = Ar n
	A timeout value of
	.Ar n
	seconds.
	.It Sy user No = Ar username
	Execute the test or test group as
	.Ar username .
	.El
	.
	.Sh OPTIONS
	.Bl -tag -width "-o outputdir"
	.It Fl c Ar runfile
	Specify a
	.Ar runfile
	to be consumed by the run command.
	.It Fl d
	Dry run mode.
	Execute no tests, but print a description of each test that would have been run.
	.It Fl m
	Enable kmemleak reporting (Linux only)
	.It Fl g
	Create test groups from any directories found while searching for tests.
	.It Fl o Ar outputdir
	Specify the directory in which to write test results.
	.It Fl p Ar script
	Run
	.Ar script
	prior to any test or test group.
	.It Fl P Ar script
	Run
	.Ar script
	after any test or test group.
	.It Fl q
	Print only the results summary to the standard output.
	.It Fl s Ar script
	Run
	.Ar script
	as a failsafe after any test is killed.
	.It Fl S Ar username
	Execute the failsafe script as
	.Ar username .
	.It Fl t Ar n
	Specify a timeout value of
	.Ar n
	seconds per test.
	.It Fl u Ar username
	Execute tests or test groups as
	.Ar username .
	.It Fl w Ar runfile
	Specify the name of the
	.Ar runfile
	to create.
	.It Fl x Ar username
	Execute the pre script as
	.Ar username .
	.It Fl X Ar username
	Execute the post script as
	.Ar username .
	.El
	.
	.Sh EXAMPLES
	.Bl -tag -width "-h"
	.It Sy Example 1 : No Running ad-hoc tests.
	This example demonstrates the simplest invocation of
	.Nm .
	.Bd -literal
	.No % Nm run Ar my-tests
	Test: /home/jkennedy/my-tests/test-01 [00:02] [PASS]
	Test: /home/jkennedy/my-tests/test-02 [00:04] [PASS]
	Test: /home/jkennedy/my-tests/test-03 [00:01] [PASS]

	Results Summary
	PASS 3

	Running Time: 00:00:07
	Percent passed: 100.0%
	Log directory: /var/tmp/test_results/20120923T180654
	.Ed
	.It Sy Example 2 : No Creating a Ar runfile No for future use.
	This example demonstrates creating a
	.Ar runfile
	with non-default options.
	.Bd -literal
	.No % Nm run Fl p Ar setup Fl x Ar root Fl g Fl w Ar new-tests.run Ar new-tests
	.No % Nm cat Pa new-tests.run
	[DEFAULT]
	pre = setup
	post_user =
	quiet = False
	user =
	timeout = 60
	post =
	pre_user = root
	outputdir = /var/tmp/test_results

	[/home/jkennedy/new-tests]
	tests = ['test-01', 'test-02', 'test-03']
	.Ed
	.El
	.
	.Sh SEE ALSO
	.Xr sudo 1m