| .\" |
| .\" This file and its contents are supplied under the terms of the |
| .\" Common Development and Distribution License ("CDDL"), version 1.0. |
| .\" You may only use this file in accordance with the terms of version |
| .\" 1.0 of the CDDL. |
| .\" |
| .\" A full copy of the text of the CDDL should have accompanied this |
| .\" source. A copy of the CDDL is also available via the Internet at |
| .\" http://www.illumos.org/license/CDDL. |
| .\" |
| .\" Copyright (c) 2012 by Delphix. All rights reserved. |
| .\" |
| .Dd May 26, 2021 |
| .Dt RUN 1 |
| .Os |
| . |
| .Sh NAME |
| .Nm run |
| .Nd find, execute, and log the results of tests |
| .Sh SYNOPSIS |
| .Nm |
| .Op Fl dgq |
| .Op Fl o Ar outputdir |
| .Op Fl pP Ar script |
| .Op Fl t seconds |
| .Op Fl uxX Ar username |
| .Ar pathname Ns No … |
| .Pp |
| .Nm |
| .Fl w Ar runfile |
| .Op Fl gq |
| .Op Fl o Ar outputdir |
| .Op Fl pP Ar script |
| .Op Fl t seconds |
| .Op Fl uxX Ar username |
| .Ar pathname Ns No … |
| .Pp |
| .Nm |
| .Fl c Ar runfile |
| .Op Fl dq |
| .Pp |
| .Nm |
| .Op Fl h |
| . |
| .Sh DESCRIPTION |
| .Nm |
| command has three basic modes of operation. |
| With neither |
| .Fl c |
| nor |
| .Fl w , |
| .Nm |
| processes the arguments provided on |
| the command line, adding them to the list for this run. |
| If a specified |
| .Ar pathname |
| is an executable file, it is added as a test. |
| If a specified |
| .Ar pathname |
| is a directory, the behavior depends upon the presence of |
| .Fl g . |
| If |
| .Fl g |
| is specified, the directory is treated as a test group. |
| See the section on |
| .Sy Test Groups |
| below. |
| Without |
| .Fl g , |
| .Nm |
| simply descends into the directory looking for executable files. |
| The tests are then executed, and the results are logged. |
| .Pp |
| With |
| .Fl w , |
| .Nm |
| finds tests in the manner described above. |
| Rather than executing the tests and logging the results, the test configuration |
| is stored in a |
| .Ar runfile , |
| which can be used in future invocations, or edited |
| to modify which tests are executed and which options are applied. |
| Options included on the command line with |
| .Fl w |
| become defaults in the |
| .Ar runfile . |
| .Pp |
| With |
| .Fl c , |
| .Nm |
| parses a |
| .Ar runfile , |
| which can specify a series of tests and test groups to be executed. |
| The tests are then executed, and the results are logged. |
| . |
| .Ss Test Groups |
| A test group is comprised of a set of executable files, all of which exist in |
| one directory. |
| The options specified on the command line or in a |
| .Ar runfile |
| apply to individual tests in the group. |
| The exception is options pertaining to pre and post scripts, which act on all tests as a group. |
| Rather than running before and after each test, |
| these scripts are run only once each at the start and end of the test group. |
| .Ss Test Execution |
| The specified tests run serially, and are typically assigned results according |
| to exit values. |
| Tests that exit zero and non-zero are marked |
| .Sy PASS |
| and |
| .Sy FAIL , |
| respectively. |
| When a pre script fails for a test group, only the post script is executed, |
| and the remaining tests are marked |
| .Sy SKIPPED . |
| Any test that exceeds |
| its |
| .Ar timeout |
| is terminated, and marked |
| .Sy KILLED . |
| .Pp |
| By default, tests are executed with the credentials of the |
| .Nm |
| script. |
| Executing tests with other credentials is done via |
| .Xr sudo 1m , |
| which must |
| be configured to allow execution without prompting for a password. |
| Environment variables from the calling shell are available to individual tests. |
| During test execution, the working directory is changed to |
| .Ar outputdir . |
| . |
| .Ss Output Logging |
| By default, |
| .Nm |
| will print one line on standard output at the conclusion |
| of each test indicating the test name, result and elapsed time. |
| Additionally, for each invocation of |
| .Nm , |
| a directory is created using the ISO 8601 date format. |
| Within this directory is a file named |
| .Sy log |
| containing all the |
| test output with timestamps, and a directory for each test. |
| Within the test directories, there is one file each for standard output, |
| standard error and merged output. |
| The default location for the |
| .Ar outputdir |
| is |
| .Pa /var/tmp/test_results . |
| .Ss "Runfiles" |
| The |
| .Ar runfile |
| is an INI-style configuration file that describes a test run. |
| The file has one section named |
| .Sy DEFAULT , |
| which contains configuration option |
| names and their values in |
| .Sy name No = Ar value |
| format. |
| The values in this section apply to all the subsequent sections, |
| unless they are also specified there, in which case the default is overridden. |
| The remaining section names are the absolute pathnames of files and directories, |
| describing tests and test groups respectively. |
| The legal option names are: |
| .Bl -tag -width "tests = ['filename', …]" |
| .It Sy outputdir No = Ar pathname |
| The name of the directory that holds test logs. |
| .It Sy pre No = Ar script |
| Run |
| .Ar script |
| prior to the test or test group. |
| .It Sy pre_user No = Ar username |
| Execute the pre script as |
| .Ar username . |
| .It Sy post No = Ar script |
| Run |
| .Ar script |
| after the test or test group. |
| .It Sy post_user No = Ar username |
| Execute the post script as |
| .Ar username . |
| .It Sy quiet No = Sy True Ns | Ns Sy False |
| If |
| .Sy True , |
| only the results summary is printed to standard out. |
| .It Sy tests No = [ Ns Ar 'filename' , No … ] |
| Specify a list of |
| .Ar filenames |
| for this test group. |
| Only the basename of the absolute path is required. |
| This option is only valid for test groups, and each |
| .Ar filename |
| must be single quoted. |
| .It Sy timeout No = Ar n |
| A timeout value of |
| .Ar n |
| seconds. |
| .It Sy user No = Ar username |
| Execute the test or test group as |
| .Ar username . |
| .El |
| . |
| .Sh OPTIONS |
| .Bl -tag -width "-o outputdir" |
| .It Fl c Ar runfile |
| Specify a |
| .Ar runfile |
| to be consumed by the run command. |
| .It Fl d |
| Dry run mode. |
| Execute no tests, but print a description of each test that would have been run. |
| .It Fl m |
| Enable kmemleak reporting (Linux only) |
| .It Fl g |
| Create test groups from any directories found while searching for tests. |
| .It Fl o Ar outputdir |
| Specify the directory in which to write test results. |
| .It Fl p Ar script |
| Run |
| .Ar script |
| prior to any test or test group. |
| .It Fl P Ar script |
| Run |
| .Ar script |
| after any test or test group. |
| .It Fl q |
| Print only the results summary to the standard output. |
| .It Fl s Ar script |
| Run |
| .Ar script |
| as a failsafe after any test is killed. |
| .It Fl S Ar username |
| Execute the failsafe script as |
| .Ar username . |
| .It Fl t Ar n |
| Specify a timeout value of |
| .Ar n |
| seconds per test. |
| .It Fl u Ar username |
| Execute tests or test groups as |
| .Ar username . |
| .It Fl w Ar runfile |
| Specify the name of the |
| .Ar runfile |
| to create. |
| .It Fl x Ar username |
| Execute the pre script as |
| .Ar username . |
| .It Fl X Ar username |
| Execute the post script as |
| .Ar username . |
| .El |
| . |
| .Sh EXAMPLES |
| .Bl -tag -width "-h" |
| .It Sy Example 1 : No Running ad-hoc tests. |
| This example demonstrates the simplest invocation of |
| .Nm . |
| .Bd -literal |
| .No % Nm run Ar my-tests |
| Test: /home/jkennedy/my-tests/test-01 [00:02] [PASS] |
| Test: /home/jkennedy/my-tests/test-02 [00:04] [PASS] |
| Test: /home/jkennedy/my-tests/test-03 [00:01] [PASS] |
| |
| Results Summary |
| PASS 3 |
| |
| Running Time: 00:00:07 |
| Percent passed: 100.0% |
| Log directory: /var/tmp/test_results/20120923T180654 |
| .Ed |
| .It Sy Example 2 : No Creating a Ar runfile No for future use. |
| This example demonstrates creating a |
| .Ar runfile |
| with non-default options. |
| .Bd -literal |
| .No % Nm run Fl p Ar setup Fl x Ar root Fl g Fl w Ar new-tests.run Ar new-tests |
| .No % Nm cat Pa new-tests.run |
| [DEFAULT] |
| pre = setup |
| post_user = |
| quiet = False |
| user = |
| timeout = 60 |
| post = |
| pre_user = root |
| outputdir = /var/tmp/test_results |
| |
| [/home/jkennedy/new-tests] |
| tests = ['test-01', 'test-02', 'test-03'] |
| .Ed |
| .El |
| . |
| .Sh SEE ALSO |
| .Xr sudo 1m |