Testing Process

The testing process for a workbench release is

  1. automated tests
  2. smoke tests
  3. hand-running system & other tests
  4. hand-testing checklist
  5. platform testing

Types of Test

The JUnit tests in Desktop AstroGridfall into 4 main different kinds:

Kind Unit Tests Integration Tests System Tests Transport Tests
Tests a single class in isolationintegration of components within the hivemind containeroperation of workbench calling remote servicesequivalent functionality available using the xmlrpc or rmi AR interface
Repeatability repeatablerepeatabledependent on remote servicesdepends
Test should be named xxxUnitTest.java xxxIntegrationTest.java xxxSystemTest.java xxxTransportTest.java
added to a suite named AllxxxUnitTests.java AllxxxIntegrationTests.java AllxxxSystemTests.java AllxxxTransportTests.java
Details tests a class in isolation, outside the hivemind / workbench framework. No external resources or network services to be called. Other dependencies should be mocked or stubbed. If the test needs to fetch a url, provide a classpath resource and pass it the url of that.exercises a class within the hivemind / workbench framework -either via its public interface, or other backdoors. No external resources or services to be called. Other dependencies should be provided by hivemind system. Use ARTestSetup.javaas a fixtureTest a class within the hivemind / workbench framework using it's public AR api where possible. Use ARTestSetup.javaas a fixtureRepeats a system or integration test but drives it using the RMI or XMLRPC interface instead of the direct-function-call interface. Implemented by subclassing an existing system or integration test and changing the connection method.

Test Organization

In the org.astrogridpackage there are top-level test suites, which provide a handy way to run a large amount of tests in one go. These can be run from within Eclipse, or from the commandline using maven.

The useful set of tests while developing. 100% pass, 5 minutes to run, 30% coverage of the source (as of 26/6/07) .Comprises -
A fast suite of all unit tests. Expect 100% pass
A fairly fast suite of all integration tests - useful for detecting hivemind errors. Expect 100% pass

warningSlightly twitchy - you may find a run occasionally fails - usually because they require login to the community service (this should be coded out).
The most useful / reliable / uptodate of the system tests. Slow, and likely to fail in places at the moment.
Exercises the transports - tends to fail often at the moment.

These top-level test suites compose the individual package-level test suites. So, when writing tests, be sure to add them to an existing package test-suite (and add newly-written test suites to an existing parent test-suite) to ensure these top-level suites remain up-to-date.

The most important one to maintain is AbsolutelyAllUnitTests , which forms the automated tests for the maven build. Maven runs this test suite after compiling the source. I hope to use AllRepeatableTests as the maven build test suite, but need to rework to remove the dependency on external services (community) first.

Occasionally, it's necessary to comment out a failing test so that the build will proceed. I try to keep this to a minimum, and mark it with a @fixmeto come back to later.

Hand Running Tests

wipAt present, the system & transport tests are liable to fail at various places, depending on various services being accessible., and take a very long time to run in their entirity. This means at the moment these tests are only useful in development - not as part of the build process.

UI testing

After some thought, I've decided not to write JUnit tests for UI code. It's hard to do, and will only catch what the tests are written to catch - not formatting problems, layout errors, etc, etc. The only approach for the UI is to hand-test.

Because of this, it's important that the non-visual parts of the UI (datastructures and controlling logic) should be factored out into classes separate from the UI, so that they can be conveniently unit tested.

Smoke Tests

After the maven build assembles the application jar (which contains the desktop code plus all libraries), it uses a stripping tool to remove unused classes and code, to reduce the download size. At the moment, this reduces the size of the full DesktopAstroGridfrom a whopping 35M to 20M. With finer tuning of the stripping tool, this size could be reduced further.

After stripping, a subset of the repeatable tests are run by maven as a Smoke testto verify that no required classes have been stripped. If the smoke test fails, the build halts.

Hand-testing Checklist.

Assuming the smoke tests succeed, the build goes on to produce an installer jar. I then go through a checklist of testing

  1. plastic - try running topcat - see if it connects to the hub. likewise for aladin.
    • check whether a notification shows up in system tray.
  2. web interface. Open in browser,
    • navigate around,
    • exercise preferences,
    • try calling a function - e.g. Webserver > getURLRoot
  3. click on the ?button, then on one of the items in main UI
    • verify the browser displays correct help.
  4. quit the application.
    • verify that aladin notifies of closure.
  5. TODO: collect some test scripts that exercise a fair portion of AR
  6. astroscope.
    1. enter position: crab, radius 0.01* expect position to be resolved to coordinate
    2. run search
    3. verify that buttons for aladin and topcat appear in LHS.
    4. check history
    5. switch to hyperbolic view (as it's still downloading).
    6. switch between degrees and sexagesimal, and observe display and inputs change
    7. switch to services tab. select a service and click the link - verify registry viewer appears for this entry.
    8. verify that there's not overmany errors in the services - should expect < 10%
    9. click on the 'tasks indicator' to see list of pending queries.
    10. back to hyperbolic view. try moving through the tree, and using the 'go to top' button
    11. do same on radial view.
    12. use 'halt' to halt query before it gets overlarge.
    13. right-click on data nodes and send them to aladin / topcat
    14. double click on nodes to select
      1. test displaying in aladin / topcat / saving by clicking on buttons on LHS
      2. test 'clear selection'
  7. Helioscope. * repeat similar to astroscope. run with default parameters.
  8. Task Runner: Complete this... 1 Loading and saving tasks 1 Query Builder 1 test authenticated access to CEA app
  9. VoExplorer: Complete this...
  10. FileExplorer: Complete...

Platform Testing

For each variant of workbench, repeat a subset of the manual testing checklist on each target operating system - Windows XP, OS X, Ubuntu Linux. For XP and Ubuntu, verify that system tray icon appears.

(I use Parallels to run XP and Ubuntu in a window on OSX)