The testing process for a workbench release is
- automated tests
- smoke tests
- hand-running system & other tests
- hand-testing checklist
- platform testing
Types of Test
The JUnit tests in Desktop
fall into 4 main different kinds:
||a single class in isolation||integration of components within the hivemind
container||operation of workbench calling remote
services||equivalent functionality available using the
xmlrpc or rmi AR interface|
||repeatable||repeatable||dependent on remote services||depends|
Test should be named
added to a suite named
||tests a class in isolation, outside the
hivemind / workbench framework. No external resources or network
services to be called. Other dependencies should be mocked or
stubbed. If the test needs to fetch a url, provide a classpath
resource and pass it the url of that.||exercises a class within the hivemind /
workbench framework -either via its public interface, or other
backdoors. No external resources or services to be called. Other
dependencies should be provided by hivemind system. Use
ARTestSetup.javaas a fixture
|Test a class within the hivemind / workbench
framework using it's public AR api where possible. Use
ARTestSetup.javaas a fixture
|Repeats a system or integration test but
drives it using the RMI or XMLRPC interface instead of the
direct-function-call interface. Implemented by subclassing an
existing system or integration test and changing the connection
package there are top-level test suites,
which provide a handy way to run a large amount of tests in one go.
These can be run from within Eclipse, or from the commandline using
- The useful set of tests while developing. 100% pass, 5 minutes
to run, 30% coverage of the source (as of 26/6/07) .Comprises -
- A fast suite of all unit tests. Expect 100% pass
- A fairly fast suite of all integration tests - useful for
detecting hivemind errors. Expect 100% pass
Slightly twitchy - you may find a run occasionally
fails - usually because they require login to the community service
(this should be coded out).
- The most useful / reliable / uptodate of the system tests.
Slow, and likely to fail in places at the moment.
- Exercises the transports - tends to fail often at the
These top-level test suites compose the individual
package-level test suites. So, when writing tests, be sure to add
them to an existing package test-suite (and add newly-written test
suites to an existing parent test-suite) to ensure these top-level
suites remain up-to-date.
The most important one to maintain is
, which forms the automated tests for the maven build. Maven
runs this test suite after compiling the source. I hope to use
as the maven build test suite, but need to rework to remove
the dependency on external services (community) first.
Occasionally, it's necessary to comment out a failing test
so that the build will proceed. I try to keep this to a minimum,
and mark it with a
to come back to later.
Hand Running Tests
At present, the system & transport tests are liable to
fail at various places, depending on various services being
accessible., and take a very long time to run in their entirity.
This means at the moment these tests are only useful in development
- not as part of the build process.
After some thought, I've decided not to write JUnit tests
for UI code. It's hard to do, and will only catch what the tests
are written to catch - not formatting problems, layout errors, etc,
etc. The only approach for the UI is to hand-test.
Because of this, it's important that the non-visual parts of
the UI (datastructures and controlling logic) should be factored
out into classes separate from the UI, so that they can be
conveniently unit tested.
After the maven build assembles the application jar (which
contains the desktop code plus all libraries), it uses a stripping
tool to remove unused classes and code, to reduce the download
size. At the moment, this reduces the size of the full
from a whopping 35M to 20M. With finer tuning
of the stripping tool, this size could be reduced further.
After stripping, a subset of the repeatable tests are run by
maven as a
to verify that no required classes have
been stripped. If the smoke test fails, the build halts.
Assuming the smoke tests succeed, the build goes on to
produce an installer jar. I then go through a checklist of testing
- plastic - try running topcat - see if it connects to the hub.
likewise for aladin.
- check whether a notification shows up in system tray.
- web interface. Open in browser,
- navigate around,
- exercise preferences,
- try calling a function - e.g.
Webserver > getURLRoot
- click on the
?button, then on one of the items in main UI
- verify the browser displays correct help.
- quit the application.
- verify that aladin notifies of closure.
- TODO: collect some test scripts that exercise a fair portion of
- enter position:
0.01* expect position to be resolved to
- run search
- verify that buttons for aladin and topcat appear in LHS.
- check history
- switch to hyperbolic view (as it's still downloading).
- switch between degrees and sexagesimal, and observe display and
- switch to services tab. select a service and click the link -
verify registry viewer appears for this entry.
- verify that there's not overmany errors in the services -
should expect < 10%
- click on the 'tasks indicator' to see list of pending
- back to hyperbolic view. try moving through the tree, and using
the 'go to top' button
- do same on radial view.
- use 'halt' to halt query before it gets overlarge.
- right-click on data nodes and send them to aladin / topcat
- double click on nodes to select
- test displaying in aladin / topcat / saving by clicking on
buttons on LHS
- test 'clear selection'
- Helioscope. * repeat similar to astroscope. run with default
- Task Runner: Complete this... 1 Loading and saving tasks 1
Query Builder 1 test authenticated access to CEA app
VoExplorer: Complete this...
For each variant of workbench, repeat a subset of the manual
testing checklist on each target operating system - Windows XP, OS
X, Ubuntu Linux. For XP and Ubuntu, verify that system tray icon
(I use Parallels to run XP and Ubuntu in a window on OSX)