appserver/tests/v2-tests/appserv-tests/devtestsNG/README

This document should be used as a guideline when creating new 
development unit tests based on TestNG.

The directory structure overall looks like this:

appserv-tests
     |
     ---- config.properties 
     |
     ---- common/
     |     |
     |     ---- common.xml
     |     |
     |     ---- properties.xml
     |     |
     |     ...(others like derby.properties etc.)
     |
     ---- devtestsNG/ 
           |
           ---- build.xml
           |
	   ---- common
	   |     |
	   |     ---- util/java/src/....
	   |     |
	   |     ---- build.xml
	   |
	   ---- mytest-module-1
	   |      |
	   |      ---- build.xml
	   |      |
	   |      ---- build.properties
	   |      |
	   |      ---- testng.xml
	   |      |
	   |      ---- tests
	   |            |
	   |            ---- com/sun/mytests/Test1.java
	   |            ...
	   |
	   |---- mytest-module-2
	   |      |
	   |      ---- build.xml
	   |      |
	   |      ---- build.properties
	   |      |
	   |      ---- my-submodule-2-1
	   |      |      |
	   |      |      ---- (replicate dir structure of module-1 or 2)
	   |      |
	   |      ---- my-submodule-2-2
	   |             |
	   |             ---- (replicate dir structure of module-1 or 2)
	   |
	   |----- ... other modules


Features:

- Ability to run test across all modules based on group membership.
- Ability to only run failed tests from previous test runs
- Ability to generate a drill-down report of the test output
- Provide an Java API and implementation to programmatically
  perform certain services like check if the server is
  running or if a certain app is deployed successfully etc.
  Essentially these features will provide the ability to check
  whether certain pre-conditions that are needed for the tests
  to be performed are satisfied are not.
  The actual tests themselves can be configured to run only if
  the preconditions are satisfied. This feature is work in progress.


Instructions:

- Env variable like APS_HOME, S1AS_HOME need to be set
  See https://glassfish.dev.java.net/public/GuidelinesandConventions.html
- The common framework facilities is extended by adding
  common target in common-build.xml
- Any module/submodule build targets linked with any of the
  targets at the top-level(appserv-tests/devtests*/) should
  have their configuration defined in the top-level
  configuration files(common/config.properties)
- Please see the README in the template directory for writing tests
  for a new module

The following directories are use by TestNG and the devtests framework
and should not used by the devtest authors directly
- test-output
- test-report
- test-tmp

New (a bunch of common targets already exist - see appserv-tests/config/common.xml)
top-level targets defined in common-build.xml
common-compile-testng-tests
common-run
common-run-groups
common-run-failed
common-report

TODO/Suggestions (in no particular order):

- Provide a list model for running the tests. The list model, as opposed
  to the current hierarchichal tree model, has a list of all the test
  directory in a single file. The advantage of using list model is that
  each of the test directories can be executed in a separate ant process
  (via a shell/batch script) and the failure of one does affect 
  the execution of others - TBD
- Reduce number of environmental variables needed to get the tests going
- Configuration parameter - documentation/clarification - so that the user can
  understand the meaning of the configuration parameters and set them correctly
- Ownership issues - designate owners for the tests for maintenance and quick resolution
- Provide default values commonly used property names
  like web.xml, sun-web.xml
  These would say default to ${basedir}/descriptors/web.xml
  and ${basedir}/descriptors/sun-web.xml respectively
- Clean up properties.xml/common.xml - remove unused targets, standardize
  target/property names
- Come up with a standardized set of group names for TestNG 
  group membership which will enable us to run tests tagged
  to specific groups across modules.
- Support running devtests running on a different machine.
  (I presume this will still require an installation of glassfish
  on the local machine since a lot of the current tests
  add ${env.S1AS_HOME}/lib/*.jar to the execution classpath)
- maven'ize the top-level devtest targets so that it
  blends in seamlessly with the rest of the glassfish
- IDE support
- Able to run a single test
- Logs
  Test Log <=> Server Log
  Both logs captured, and marked for each test
- Remote Instances
  PE-EE flexibility
- Persistence EE and Java SE containers
- Roadmap should include Tools, i.e. NetBeans integration
- Deployment, jars support
  Java EE technology and Non-Java EE Extensions (i.e. MBean, etc)
- Able to run old framework

For questions/comments email: quality@glassfish.dev.java.net