| <!DOCTYPE html> | |
| <html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang=""> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="generator" content="pandoc" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" /> | |
| <title>Native/Unit Test Development Guidelines</title> | |
| <style> | |
| code{white-space: pre-wrap;} | |
| span.smallcaps{font-variant: small-caps;} | |
| div.columns{display: flex; gap: min(4vw, 1.5em);} | |
| div.column{flex: auto; overflow-x: auto;} | |
| div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} | |
| ul.task-list{list-style: none;} | |
| ul.task-list li input[type="checkbox"] { | |
| width: 0.8em; | |
| margin: 0 0.8em 0.2em -1.6em; | |
| vertical-align: middle; | |
| } | |
| .display.math{display: block; text-align: center; margin: 0.5rem auto;} | |
| </style> | |
| <link rel="stylesheet" href="../make/data/docs-resources/resources/jdk-default.css" /> | |
| <!--[if lt IE 9]> | |
| <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script> | |
| <![endif]--> | |
| </head> | |
| <body> | |
| <header id="title-block-header"> | |
| <h1 class="title">Native/Unit Test Development Guidelines</h1> | |
| </header> | |
| <nav id="TOC" role="doc-toc"> | |
| <ul> | |
| <li><a href="#good-test-properties" id="toc-good-test-properties">Good | |
| test properties</a> | |
| <ul> | |
| <li><a href="#lightness" id="toc-lightness">Lightness</a></li> | |
| <li><a href="#isolation" id="toc-isolation">Isolation</a></li> | |
| <li><a href="#atomicity-and-self-containment" | |
| id="toc-atomicity-and-self-containment">Atomicity and | |
| self-containment</a></li> | |
| <li><a href="#repeatability" | |
| id="toc-repeatability">Repeatability</a></li> | |
| <li><a href="#informativeness" | |
| id="toc-informativeness">Informativeness</a></li> | |
| <li><a href="#testing-instead-of-visiting" | |
| id="toc-testing-instead-of-visiting">Testing instead of | |
| visiting</a></li> | |
| <li><a href="#nearness" id="toc-nearness">Nearness</a></li> | |
| </ul></li> | |
| <li><a href="#asserts" id="toc-asserts">Asserts</a> | |
| <ul> | |
| <li><a href="#several-checks" id="toc-several-checks">Several | |
| checks</a></li> | |
| <li><a href="#first-parameter-is-expected-value" | |
| id="toc-first-parameter-is-expected-value">First parameter is expected | |
| value</a></li> | |
| <li><a href="#floating-point-comparison" | |
| id="toc-floating-point-comparison">Floating-point comparison</a></li> | |
| <li><a href="#c-string-comparison" id="toc-c-string-comparison">C string | |
| comparison</a></li> | |
| <li><a href="#error-messages" id="toc-error-messages">Error | |
| messages</a></li> | |
| <li><a href="#uncluttered-output" | |
| id="toc-uncluttered-output">Uncluttered output</a></li> | |
| <li><a href="#failures-propagation" | |
| id="toc-failures-propagation">Failures propagation</a></li> | |
| </ul></li> | |
| <li><a href="#naming-and-grouping" id="toc-naming-and-grouping">Naming | |
| and Grouping</a> | |
| <ul> | |
| <li><a href="#test-group-names" id="toc-test-group-names">Test group | |
| names</a></li> | |
| <li><a href="#filename" id="toc-filename">Filename</a></li> | |
| <li><a href="#file-location" id="toc-file-location">File | |
| location</a></li> | |
| <li><a href="#test-names" id="toc-test-names">Test names</a></li> | |
| <li><a href="#fixture-classes" id="toc-fixture-classes">Fixture | |
| classes</a></li> | |
| <li><a href="#friend-classes" id="toc-friend-classes">Friend | |
| classes</a></li> | |
| <li><a href="#oscpu-specific-tests" id="toc-oscpu-specific-tests">OS/CPU | |
| specific tests</a></li> | |
| </ul></li> | |
| <li><a href="#miscellaneous" id="toc-miscellaneous">Miscellaneous</a> | |
| <ul> | |
| <li><a href="#hotspot-style" id="toc-hotspot-style">Hotspot | |
| style</a></li> | |
| <li><a href="#codetest-metrics" id="toc-codetest-metrics">Code/test | |
| metrics</a></li> | |
| <li><a href="#access-to-non-public-members" | |
| id="toc-access-to-non-public-members">Access to non-public | |
| members</a></li> | |
| <li><a href="#death-tests" id="toc-death-tests">Death tests</a></li> | |
| <li><a href="#external-flags" id="toc-external-flags">External | |
| flags</a></li> | |
| <li><a href="#test-specific-flags" | |
| id="toc-test-specific-flags">Test-specific flags</a></li> | |
| <li><a href="#flag-restoring" id="toc-flag-restoring">Flag | |
| restoring</a></li> | |
| <li><a href="#googletest-documentation" | |
| id="toc-googletest-documentation">GoogleTest documentation</a></li> | |
| </ul></li> | |
| <li><a href="#todo" id="toc-todo">TODO</a></li> | |
| </ul> | |
| </nav> | |
| <p>The purpose of these guidelines is to establish a shared vision on | |
| what kind of native tests and how we want to develop them for Hotspot | |
| using GoogleTest. Hence these guidelines include style items as well as | |
| test approach items.</p> | |
| <p>First section of this document describes properties of good tests | |
| which are common for almost all types of test regardless of language, | |
| framework, etc. Further sections provide recommendations to achieve | |
| those properties and other HotSpot and/or GoogleTest specific | |
| guidelines.</p> | |
| <h2 id="good-test-properties">Good test properties</h2> | |
| <h3 id="lightness">Lightness</h3> | |
| <p>Use the most lightweight type of tests.</p> | |
| <p>In Hotspot, there are 3 different types of tests regarding their | |
| dependency on a JVM, each next level is slower than previous</p> | |
| <ul> | |
| <li><p><code>TEST</code> : a test does not depend on a JVM</p></li> | |
| <li><p><code>TEST_VM</code> : a test does depend on an initialized JVM, | |
| but are supposed not to break a JVM, i.e. leave it in a workable | |
| state.</p></li> | |
| <li><p><code>TEST_OTHER_VM</code> : a test depends on a JVM and requires | |
| a freshly initialized JVM or leaves a JVM in non-workable state</p></li> | |
| </ul> | |
| <h3 id="isolation">Isolation</h3> | |
| <p>Tests have to be isolated: not to have visible side-effects, | |
| influences on other tests results.</p> | |
| <p>Results of one test should not depend on test execution order, other | |
| tests, otherwise it is becoming almost impossible to find out why a test | |
| failed. Due to hotspot-specific, it is not so easy to get a full | |
| isolation, e.g. we share an initialized JVM between all | |
| <code>TEST_VM</code> tests, so if your test changes JVM's state too | |
| drastically and does not change it back, you had better consider | |
| <code>TEST_OTHER_VM</code>.</p> | |
| <h3 id="atomicity-and-self-containment">Atomicity and | |
| self-containment</h3> | |
| <p>Tests should be <em>atomic</em> and <em>self-contained</em> at the | |
| same time.</p> | |
| <p>One test should check a particular part of a class, subsystem, | |
| functionality, etc. Then it is quite easy to determine what parts of a | |
| product are broken basing on test failures. On the other hand, a test | |
| should test that part more-or-less entirely, because when one sees a | |
| test <code>FooTest::bar</code>, they assume all aspects of bar from | |
| <code>Foo</code> are tested.</p> | |
| <p>However, it is impossible to cover all aspects even of a method, not | |
| to mention a subsystem. In such cases, it is recommended to have several | |
| tests, one for each aspect of a thing under test. For example one test | |
| to tests how <code>Foo::bar</code> works if an argument is | |
| <code>null</code>, another test to test how it works if an argument is | |
| acceptable but <code>Foo</code> is not in the right state to accept it | |
| and so on. This helps not only to make tests atomic, self-contained but | |
| also makes test name self-descriptive (discussed in more details in <a | |
| href="#test-names">Test names</a>).</p> | |
| <h3 id="repeatability">Repeatability</h3> | |
| <p>Tests have to be repeatable.</p> | |
| <p>Reproducibility is very crucial for a test. No one likes sporadic | |
| test failures, they are hard to investigate, fix and verify a fix.</p> | |
| <p>In some cases, it is quite hard to write a 100% repeatable test, | |
| since besides a test there can be other moving parts, e.g. in case of | |
| <code>TEST_VM</code> there are several concurrently running threads. | |
| Despite this, we should try to make a test as reproducible as | |
| possible.</p> | |
| <h3 id="informativeness">Informativeness</h3> | |
| <p>In case of a failure, a test should be as <em>informative</em> as | |
| possible.</p> | |
| <p>Having more information about a test failure than just compared | |
| values can be very useful for failure troubleshooting, it can reduce or | |
| even completely eliminate debugging hours. This is even more important | |
| in case of not 100% reproducible failures.</p> | |
| <p>Achieving this property, one can easily make a test too verbose, so | |
| it will be really hard to find useful information in the ocean of | |
| useless information. Hence they should not only think about how to | |
| provide <a href="#error-messages">good information</a>, but also <a | |
| href="#uncluttered-output">when to do it</a>.</p> | |
| <h3 id="testing-instead-of-visiting">Testing instead of visiting</h3> | |
| <p>Tests should <em>test</em>.</p> | |
| <p>It is not enough just to "visit" some code, a test should check that | |
| code does that it has to do, compare return values with expected values, | |
| check that desired side effects are done, and undesired are not, and so | |
| on. In other words, a test should contain at least one GoogleTest | |
| assertion and do not rely on JVM asserts.</p> | |
| <p>Generally speaking to write a good test, one should create a model of | |
| the system under tests, a model of possible bugs (or bugs which one | |
| wants to find) and design tests using those models.</p> | |
| <h3 id="nearness">Nearness</h3> | |
| <p>Prefer having checks inside test code.</p> | |
| <p>Not only does having test logic outside, e.g. verification method, | |
| depending on asserts in product code contradict with several items above | |
| but also decreases test's readability and stability. It is much easier | |
| to understand that a test is testing when all testing logic is located | |
| inside a test or nearby in shared test libraries. As a rule of thumb, | |
| the closer a check to a test, the better.</p> | |
| <h2 id="asserts">Asserts</h2> | |
| <h3 id="several-checks">Several checks</h3> | |
| <p>Prefer <code>EXPECT</code> over <code>ASSERT</code> if possible.</p> | |
| <p>This is related to the <a href="#informativeness">informativeness</a> | |
| property of tests, information for other checks can help to better | |
| localize a defect's root-cause. One should use <code>ASSERT</code> if it | |
| is impossible to continue test execution or if it does not make much | |
| sense. Later in the text, <code>EXPECT</code> forms will be used to | |
| refer to both <code>ASSERT/EXPECT</code>.</p> | |
| <p>When it is possible to make several different checks, but impossible | |
| to continue test execution if at least one check fails, you can use | |
| <code>::testing::Test::HasNonfatalFailure()</code> function. The | |
| recommended way to express that is | |
| <code>ASSERT_FALSE(::testing::Test::HasNonfatalFailure())</code>. | |
| Besides making it clear why a test is aborted, it also allows you to | |
| provide more information about a failure.</p> | |
| <h3 id="first-parameter-is-expected-value">First parameter is expected | |
| value</h3> | |
| <p>In all equality assertions, expected values should be passed as the | |
| first parameter.</p> | |
| <p>This convention is adopted by GoogleTest, and there is a slight | |
| difference in how GoogleTest treats parameters, the most important one | |
| is <code>null</code> detection. Due to different reasons, | |
| <code>null</code> detection is enabled only for the first parameter, | |
| that is to said <code>EXPECT_EQ(NULL, object)</code> checks that object | |
| is <code>null</code>, while <code>EXPECT_EQ(object, NULL)</code> checks | |
| that object equals to <code>NULL</code>, GoogleTest is very strict | |
| regarding types of compared values so the latter will generates a | |
| compile-time error.</p> | |
| <h3 id="floating-point-comparison">Floating-point comparison</h3> | |
| <p>Use floating-point special macros to compare | |
| <code>float/double</code> values.</p> | |
| <p>Because of floating-point number representations and round-off | |
| errors, regular equality comparison will not return true in most cases. | |
| There are special <code>EXPECT_FLOAT_EQ/EXPECT_DOUBLE_EQ</code> | |
| assertions which check that the distance between compared values is not | |
| more than 4 ULPs, there is also <code>EXPECT_NEAR(v1, v2, eps)</code> | |
| which checks that the absolute value of the difference between | |
| <code>v1</code> and <code>v2</code> is not greater than | |
| <code>eps</code>.</p> | |
| <h3 id="c-string-comparison">C string comparison</h3> | |
| <p>Use string special macros for C strings comparisons.</p> | |
| <p><code>EXPECT_EQ</code> just compares pointers' values, which is | |
| hardly what one wants comparing C strings. GoogleTest provides | |
| <code>EXPECT_STREQ</code> and <code>EXPECT_STRNE</code> macros to | |
| compare C string contents. There are also case-insensitive versions | |
| <code>EXPECT_STRCASEEQ</code>, <code>EXPECT_STRCASENE</code>.</p> | |
| <h3 id="error-messages">Error messages</h3> | |
| <p>Provide informative, but not too verbose error messages.</p> | |
| <p>All GoogleTest asserts print compared expressions and their values, | |
| so there is no need to have them in error messages. Asserts print only | |
| compared values, they do not print any of interim variables, e.g. | |
| <code>ASSERT_TRUE((val1 == val2 && isFail(foo(8))) || i == 18)</code> | |
| prints only one value. If you use some complex predicates, please | |
| consider <code>EXPECT_PRED*</code> or <code>EXPECT_FORMAT_PRED</code> | |
| assertions family, they check that a predicate returns true/success and | |
| print out all parameters values.</p> | |
| <p>However in some cases, default information is not enough, a commonly | |
| used example is an assert inside a loop, GoogleTest will not print | |
| iteration values (unless it is an assert's parameter). Other | |
| demonstrative examples are printing error code and a corresponding error | |
| message; printing internal states which might have an impact on results. | |
| One should add this information to assert message using | |
| <code><<</code> operator.</p> | |
| <h3 id="uncluttered-output">Uncluttered output</h3> | |
| <p>Print information only if it is needed.</p> | |
| <p>Too verbose tests which print all information even if they pass are | |
| very bad practice. They just pollute output, so it becomes harder to | |
| find useful information. In order not print information till it is | |
| really needed, one should consider saving it to a temporary buffer and | |
| pass to an assert. <a | |
| href="https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/shared/test_memset_with_concurrent_readers.cpp" | |
| class="uri">https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/shared/test_memset_with_concurrent_readers.cpp</a> | |
| has a good example how to do that.</p> | |
| <h3 id="failures-propagation">Failures propagation</h3> | |
| <p>Wrap a subroutine call into <code>EXPECT_NO_FATAL_FAILURE</code> | |
| macro to propagate failures.</p> | |
| <p><code>ASSERT</code> and <code>FAIL</code> abort only the current | |
| function, so if you have them in a subroutine, a test will not be | |
| aborted after the subroutine even if <code>ASSERT</code> or | |
| <code>FAIL</code> fails. You should call such subroutines in | |
| <code>ASSERT_NO_FATAL_FAILURE</code> macro to propagate fatal failures | |
| and abort a test. <code>(EXPECT|ASSERT)_NO_FATAL_FAILURE</code> can also | |
| be used to provide more information.</p> | |
| <p>Due to obvious reasons, there are no | |
| <code>(EXPECT|ASSERT)_NO_NONFATAL_FAILURE</code> macros. However, if you | |
| need to check if a subroutine generated a nonfatal failure (failed an | |
| <code>EXPECT</code>), you can use | |
| <code>::testing::Test::HasNonfatalFailure</code> function, or | |
| <code>::testing::Test::HasFailure</code> function to check if a | |
| subroutine generated any failures, see <a href="#several-checks">Several | |
| checks</a>.</p> | |
| <h2 id="naming-and-grouping">Naming and Grouping</h2> | |
| <h3 id="test-group-names">Test group names</h3> | |
| <p>Test group names should be in CamelCase, start and end with a letter. | |
| A test group should be named after tested class, functionality, | |
| subsystem, etc.</p> | |
| <p>This naming scheme helps to find tests, filter them and simplifies | |
| test failure analysis. For example, class <code>Foo</code> - test group | |
| <code>Foo</code>, compiler logging subsystem - test group | |
| <code>CompilerLogging</code>, G1 GC - test group <code>G1GC</code>, and | |
| so forth.</p> | |
| <h3 id="filename">Filename</h3> | |
| <p>A test file must have <code>test_</code> prefix and <code>.cpp</code> | |
| suffix.</p> | |
| <p>Both are actually requirements from the current build system to | |
| recognize your tests.</p> | |
| <h3 id="file-location">File location</h3> | |
| <p>Test file location should reflect a location of the tested part of | |
| the product.</p> | |
| <ul> | |
| <li><p>All unit tests for a class from <code>foo/bar/baz.cpp</code> | |
| should be placed <code>foo/bar/test_baz.cpp</code> in | |
| <code>hotspot/test/native/</code> directory. Having all tests for a | |
| class in one file is a common practice for unit tests, it helps to see | |
| all existing tests at once, share functions and/or resources without | |
| losing encapsulation.</p></li> | |
| <li><p>For tests which test more than one class, directory hierarchy | |
| should be the same as product hierarchy, and file name should reflect | |
| the name of the tested subsystem/functionality. For example, if a | |
| sub-system under tests belongs to <code>gc/g1</code>, tests should be | |
| placed in <code>gc/g1</code> directory.</p></li> | |
| </ul> | |
| <p>Please note that framework prepends directory name to a test group | |
| name. For example, if <code>TEST(foo, check_this)</code> and | |
| <code>TEST(bar, check_that)</code> are defined in | |
| <code>hotspot/test/native/gc/shared/test_foo.cpp</code> file, they will | |
| be reported as <code>gc/shared/foo::check_this</code> and | |
| <code>gc/shared/bar::check_that</code>.</p> | |
| <h3 id="test-names">Test names</h3> | |
| <p>Test names should be in small_snake_case, start and end with a | |
| letter. A test name should reflect that a test checks.</p> | |
| <p>Such naming makes tests self-descriptive and helps a lot during the | |
| whole test life cycle. It is easy to do test planning, test inventory, | |
| to see what things are not tested, to review tests, to analyze test | |
| failures, to evolve a test, etc. For example | |
| <code>foo_return_0_if_name_is_null</code> is better than | |
| <code>foo_sanity</code> or <code>foo_basic</code> or just | |
| <code>foo</code>, | |
| <code>humongous_objects_can_not_be_moved_by_young_gc</code> is better | |
| than <code>ho_young_gc</code>.</p> | |
| <p>Actually using underscore is against GoogleTest project convention, | |
| because it can lead to illegal identifiers, however, this is too strict. | |
| Restricting usage of underscore for test names only and prohibiting test | |
| name starts or ends with an underscore are enough to be safe.</p> | |
| <h3 id="fixture-classes">Fixture classes</h3> | |
| <p>Fixture classes should be named after tested classes, subsystems, etc | |
| (follow <a href="#test-group-names">Test group names rule</a>) and have | |
| <code>Test</code> suffix to prevent class name conflicts.</p> | |
| <h3 id="friend-classes">Friend classes</h3> | |
| <p>All test purpose friends should have either <code>Test</code> or | |
| <code>Testable</code> suffix.</p> | |
| <p>It greatly simplifies understanding of friendship's purpose and | |
| allows statically check that private members are not exposed | |
| unexpectedly. Having <code>FooTest</code> as a friend of | |
| <code>Foo</code> without any comments will be understood as a necessary | |
| evil to get testability.</p> | |
| <h3 id="oscpu-specific-tests">OS/CPU specific tests</h3> | |
| <p>Guard OS/CPU specific tests by <code>#ifdef</code> and have OS/CPU | |
| name in filename.</p> | |
| <p>For the time being, we do not support separate directories for OS, | |
| CPU, OS-CPU specific tests, in case we will have lots of such tests, we | |
| will change directory layout and build system to support that in the | |
| same way it is done in hotspot.</p> | |
| <h2 id="miscellaneous">Miscellaneous</h2> | |
| <h3 id="hotspot-style">Hotspot style</h3> | |
| <p>Abide the norms and rules accepted in Hotspot style guide.</p> | |
| <p>Tests are a part of Hotspot, so everything (if applicable) we use for | |
| Hotspot, should be used for tests as well. Those guidelines cover | |
| test-specific things.</p> | |
| <h3 id="codetest-metrics">Code/test metrics</h3> | |
| <p>Coverage information and other code/test metrics are quite useful to | |
| decide what tests should be written, what tests should be improved and | |
| what can be removed.</p> | |
| <p>For unit tests, widely used and well-known coverage metric is branch | |
| coverage, which provides good quality of tests with relatively easy test | |
| development process. For other levels of testing, branch coverage is not | |
| as good, and one should consider others metrics, e.g. transaction flow | |
| coverage, data flow coverage.</p> | |
| <h3 id="access-to-non-public-members">Access to non-public members</h3> | |
| <p>Use explicit friend class to get access to non-public members.</p> | |
| <p>We do not use GoogleTest macro to declare friendship relation, | |
| because, from our point of view, it is less clear than an explicit | |
| declaration.</p> | |
| <p>Declaring a test fixture class as a friend class of a tested test is | |
| the easiest and the clearest way to get access. However, it has some | |
| disadvantages, here is some of them:</p> | |
| <ul> | |
| <li>Each test has to be declared as a friend</li> | |
| <li>Subclasses do not inheritance friendship relation</li> | |
| </ul> | |
| <p>In other words, it is harder to share code between tests. Hence if | |
| you want to share code or expect it to be useful in other tests, you | |
| should consider making members in a tested class protected and introduce | |
| a shared test-only class which expose those members via public | |
| functions, or even making members publicly accessible right away in a | |
| product class. If it is not an option to change members visibility, one | |
| can create a friend class which exposes members.</p> | |
| <h3 id="death-tests">Death tests</h3> | |
| <p>You can not use death tests inside <code>TEST_OTHER_VM</code> and | |
| <code>TEST_VM_ASSERT*</code>.</p> | |
| <p>We tried to make Hotspot-GoogleTest integration as transparent as | |
| possible, however, due to the current implementation of | |
| <code>TEST_OTHER_VM</code> and <code>TEST_VM_ASSERT*</code> tests, you | |
| cannot use death test functionality in them. These tests are implemented | |
| as GoogleTest death tests, and GoogleTest does not allow to have a death | |
| test inside another death test.</p> | |
| <h3 id="external-flags">External flags</h3> | |
| <p>Passing external flags to a tested JVM is not supported.</p> | |
| <p>The rationality of such design decision is to simplify both tests and | |
| a test framework and to avoid failures related to incompatible flags | |
| combination till there is a good solution for that. However there are | |
| cases when one wants to test a JVM with specific flags combination, | |
| <code>_JAVA_OPTIONS</code> environment variable can be used to do that. | |
| Flags from <code>_JAVA_OPTIONS</code> will be used in | |
| <code>TEST_VM</code>, <code>TEST_OTHER_VM</code> and | |
| <code>TEST_VM_ASSERT*</code> tests.</p> | |
| <h3 id="test-specific-flags">Test-specific flags</h3> | |
| <p>Passing flags to a tested JVM in <code>TEST_OTHER_VM</code> and | |
| <code>TEST_VM_ASSERT*</code> should be possible, but is not implemented | |
| yet.</p> | |
| <p>Facility to pass test-specific flags is needed for system, regression | |
| or other types of tests which require a fully initialized JVM in some | |
| particular configuration, e.g. with Serial GC selected. There is no | |
| support for such tests now, however, there is a plan to add that in | |
| upcoming releases.</p> | |
| <p>For now, if a test depends on flags values, it should have | |
| <code>if (!<flag>) { return }</code> guards in the very beginning | |
| and <code>@requires</code> comment similar to jtreg | |
| <code>@requires</code> directive right before test macros. <a | |
| href="https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/g1/test_g1IHOPControl.cpp" | |
| class="uri">https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/g1/test_g1IHOPControl.cpp</a> | |
| ha an example of this temporary workaround. It is important to follow | |
| that pattern as it allows us to easily find all such tests and update | |
| them as soon as there is an implementation of flag passing facility.</p> | |
| <p>In long-term, we expect jtreg to support GoogleTest tests as first | |
| class citizens, that is to say, jtreg will parse <span class="citation" | |
| data-cites="requires">@requires</span> comments and filter out | |
| inapplicable tests.</p> | |
| <h3 id="flag-restoring">Flag restoring</h3> | |
| <p>Restore changed flags.</p> | |
| <p>It is quite common for tests to configure JVM in a certain way | |
| changing flags' values. GoogleTest provides two ways to set up | |
| environment before a test and restore it afterward: using either | |
| constructor and destructor or <code>SetUp</code> and | |
| <code>TearDown</code> functions. Both ways require to use a test fixture | |
| class, which sometimes is too wordy. The simpler facilities like | |
| <code>FLAG_GUARD</code> macro or <code>*FlagSetting</code> classes could | |
| be used in such cases to restore/set values.</p> | |
| <p>Caveats:</p> | |
| <ul> | |
| <li><p>Changing a flag's value could break the invariants between flags' | |
| values and hence could lead to unexpected/unsupported JVM | |
| state.</p></li> | |
| <li><p><code>FLAG_SET_*</code> macros can change more than one flag (in | |
| order to maintain invariants) so it is hard to predict what flags will | |
| be changed and it makes restoring all changed flags a nontrivial task. | |
| Thus in case one uses <code>FLAG_SET_*</code> macros, they should use | |
| <code>TEST_OTHER_VM</code> test type.</p></li> | |
| </ul> | |
| <h3 id="googletest-documentation">GoogleTest documentation</h3> | |
| <p>In case you have any questions regarding GoogleTest itself, its | |
| asserts, test declaration macros, other macros, etc, please consult its | |
| documentation.</p> | |
| <h2 id="todo">TODO</h2> | |
| <p>Although this document provides guidelines on the most important | |
| parts of test development using GTest, it still misses a few items:</p> | |
| <ul> | |
| <li><p>Examples, esp for <a href="#access-to-non-public-members">access | |
| to non-public members</a></p></li> | |
| <li><p>test types: purpose, drawbacks, limitation</p> | |
| <ul> | |
| <li><code>TEST_VM</code></li> | |
| <li><code>TEST_VM_F</code></li> | |
| <li><code>TEST_OTHER_VM</code></li> | |
| <li><code>TEST_VM_ASSERT</code></li> | |
| <li><code>TEST_VM_ASSERT_MSG</code></li> | |
| </ul></li> | |
| <li><p>Miscellaneous</p> | |
| <ul> | |
| <li>Test libraries | |
| <ul> | |
| <li>where to place</li> | |
| <li>how to write</li> | |
| <li>how to use</li> | |
| </ul></li> | |
| <li>test your tests | |
| <ul> | |
| <li>how to run tests in random order</li> | |
| <li>how to run only specific tests</li> | |
| <li>how to run each test separately</li> | |
| <li>check that a test can find bugs it is supposed to by introducing | |
| them</li> | |
| </ul></li> | |
| <li>mocks/stubs/dependency injection</li> | |
| <li>setUp/tearDown | |
| <ul> | |
| <li>vs c-tor/d-tor</li> | |
| <li>empty test to test them</li> | |
| </ul></li> | |
| <li>internal (declared in .cpp) struct/classes</li> | |
| </ul></li> | |
| </ul> | |
| </body> | |
| </html> |