Hi all,
As we have all known for quite some time, running the full set of geoserver unit and integration tests takes quite some time. You also may have noticed that post security changes the tests take even longer on trunk. I have been spending spare cycles recently looking into how to remedy this. Here are some thoughts/findings.
- “Migration” is expensive
Our integration test setup is still using the pre 2.0 data directory structure. There are some hacks in the geoserver loading that prevent the conversion to the 2.x style that most of the tests use. Tests that want use the new setup have the option to set a flag to do so. In this case the old structure is still written out but once the application context is started the conversion takes place.
Post security changes, an additional migration takes place that sets up all the new security services. This is actually quite expensive and doing it during every test setup / tear down (and by every i mean every “one time” setup/teardown) burns through a lot of time.
- Not enough utilization of one time setup
Even without the price paid by the security migration tests still take a long time to run. The worst offenders appear to be the the wfs and restconfig modules and the reason being many of the test cases in that module don’t use one time setup because they actually modify data and configuration. And in order to ensure a fresh copy of data/configuration on each test case it forgoes one time setup and does the setup every time.
So, how do we go about fixing this.
(1) Keeping the same pattern as the existing test setup, we could simply serialize out the new structure rather than the old structure. Should be relatively straight forward and can probably use the facilities in XStreamPersister to do so. An alternative would be to have a pre-canned data directory and copy/unpack it when we have to do test setup.
(2) This is the interesting/tricky one. If tests that actually modify data are going to utilize a one-time setup then what we need is way for a test case to reload or refresh single layers at a time, returning both configuration and data to original state. Or in cases where new layers are added, remove them. Or if existing layers deleted, restore them. All of the above I believe should really just amount to some utility classes on the base test classes. However it does add a burden on the test writer since they now have to worry about initialization/cleanup of state. Also there is the issue of test order. The order of tests run in an IDE is not always the same as running from maven and since state is maintained across test methods i can see this becoming a mess in which failures occur with maven but not from the ide.
Anyways, as an exercise I decided to locally port all the restconfig tests to a one-time setup. As expected the results are significant. Currently on trunk the full restconfig test run takes approximately 5 minutes on my local machine. Utilizing the one time setup this drops down to about 1 minute 20 seconds.
So can we find a middle ground between using a one-time setup and making test writers not have to worry too much about state between test methods? One of the ideas i have is to use annotations to help with this. With Junit4 we essentially have the ability to define our own custom annotations, called “rules”. So we could do something like this:
@Test
@PostReload
public void testFoo() {
…
}
The “PostReload” annotation would essentially tell the framework that the test modifies configuration/data and after the test is run the configuration must be reloaded. This would allow tests mixing test methods that are read only with ones that modify configuration only pay the price when things are changed. Unfortunately this could still result in a lot of configuration reloads for a single test class, like in the case for restconfig.
Pushing the idea a bit further, i had the idea of being more granular about it. Doing something like this:
@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {
…
}
So instead of reloading the entire setup only a single store/layer/etc… would have to be reloaded, or deleted, etc…
Anyways, just a rough idea at this point. Adopting helper annotations like this would imply an upgrade of Junit to version 4. Which some tests are actually already using afaik. There is also TestNG, which from a customization point of view is much more flexible than Junit 4. Downside is it is really a completely different framework, the upgrade would be much less seamless and it has less built in support from our current maven and eclipse tooling.
So some food for thought at this point. Interested in hearing peoples thoughts on this.
-Justin
–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.