[Geoserver-devel] improving test times

Hi all,

As we have all known for quite some time, running the full set of geoserver unit and integration tests takes quite some time. You also may have noticed that post security changes the tests take even longer on trunk. I have been spending spare cycles recently looking into how to remedy this. Here are some thoughts/findings.

  1. “Migration” is expensive

Our integration test setup is still using the pre 2.0 data directory structure. There are some hacks in the geoserver loading that prevent the conversion to the 2.x style that most of the tests use. Tests that want use the new setup have the option to set a flag to do so. In this case the old structure is still written out but once the application context is started the conversion takes place.

Post security changes, an additional migration takes place that sets up all the new security services. This is actually quite expensive and doing it during every test setup / tear down (and by every i mean every “one time” setup/teardown) burns through a lot of time.

  1. Not enough utilization of one time setup

Even without the price paid by the security migration tests still take a long time to run. The worst offenders appear to be the the wfs and restconfig modules and the reason being many of the test cases in that module don’t use one time setup because they actually modify data and configuration. And in order to ensure a fresh copy of data/configuration on each test case it forgoes one time setup and does the setup every time.

So, how do we go about fixing this.

(1) Keeping the same pattern as the existing test setup, we could simply serialize out the new structure rather than the old structure. Should be relatively straight forward and can probably use the facilities in XStreamPersister to do so. An alternative would be to have a pre-canned data directory and copy/unpack it when we have to do test setup.

(2) This is the interesting/tricky one. If tests that actually modify data are going to utilize a one-time setup then what we need is way for a test case to reload or refresh single layers at a time, returning both configuration and data to original state. Or in cases where new layers are added, remove them. Or if existing layers deleted, restore them. All of the above I believe should really just amount to some utility classes on the base test classes. However it does add a burden on the test writer since they now have to worry about initialization/cleanup of state. Also there is the issue of test order. The order of tests run in an IDE is not always the same as running from maven and since state is maintained across test methods i can see this becoming a mess in which failures occur with maven but not from the ide.

Anyways, as an exercise I decided to locally port all the restconfig tests to a one-time setup. As expected the results are significant. Currently on trunk the full restconfig test run takes approximately 5 minutes on my local machine. Utilizing the one time setup this drops down to about 1 minute 20 seconds.

So can we find a middle ground between using a one-time setup and making test writers not have to worry too much about state between test methods? One of the ideas i have is to use annotations to help with this. With Junit4 we essentially have the ability to define our own custom annotations, called “rules”. So we could do something like this:

@Test
@PostReload
public void testFoo() {

}

The “PostReload” annotation would essentially tell the framework that the test modifies configuration/data and after the test is run the configuration must be reloaded. This would allow tests mixing test methods that are read only with ones that modify configuration only pay the price when things are changed. Unfortunately this could still result in a lot of configuration reloads for a single test class, like in the case for restconfig.

Pushing the idea a bit further, i had the idea of being more granular about it. Doing something like this:

@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {

}

So instead of reloading the entire setup only a single store/layer/etc… would have to be reloaded, or deleted, etc…

Anyways, just a rough idea at this point. Adopting helper annotations like this would imply an upgrade of Junit to version 4. Which some tests are actually already using afaik. There is also TestNG, which from a customization point of view is much more flexible than Junit 4. Downside is it is really a completely different framework, the upgrade would be much less seamless and it has less built in support from our current maven and eclipse tooling.

So some food for thought at this point. Interested in hearing peoples thoughts on this.

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

On Fri, Jun 15, 2012 at 4:38 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

(1) Keeping the same pattern as the existing test setup, we could simply
serialize out the new structure rather than the old structure. Should be
relatively straight forward and can probably use the facilities in
XStreamPersister to do so. An alternative would be to have a pre-canned data
directory and copy/unpack it when we have to do test setup.

The precanned one would be good for the tests that use the standard
mock data set, which is, I guess, the majority, so it sounds like a good idea.

However some setup only a handlful of types skipping the whole data dir
setup, so we need to be able to recognize if a test is setting up the base
data dir or not. Many tests add on top of it, but that's something we can
handle after having setup the precanned one right?

(2) This is the interesting/tricky one. If tests that actually modify data
are going to utilize a one-time setup then what we need is way for a test
case to reload or refresh single layers at a time, returning both
configuration and data to original state. Or in cases where new layers are
added, remove them. Or if existing layers deleted, restore them. All of the
above I believe should really just amount to some utility classes on the
base test classes. However it does add a burden on the test writer since
they now have to worry about initialization/cleanup of state. Also there is
the issue of test order. The order of tests run in an IDE is not always the
same as running from maven and since state is maintained across test methods
i can see this becoming a mess in which failures occur with maven but not
from the ide.

Anyways, as an exercise I decided to locally port all the restconfig tests
to a one-time setup. As expected the results are significant. Currently on
trunk the full restconfig test run takes approximately 5 minutes on my local
machine. Utilizing the one time setup this drops down to about 1 minute 20
seconds.

So can we find a middle ground between using a one-time setup and making
test writers not have to worry too much about state between test methods?
One of the ideas i have is to use annotations to help with this. With Junit4
we essentially have the ability to define our own custom annotations, called
"rules". So we could do something like this:

@Test
@PostReload
public void testFoo() {
...
}

The "PostReload" annotation would essentially tell the framework that the
test modifies configuration/data and after the test is run the configuration
must be reloaded. This would allow tests mixing test methods that are read
only with ones that modify configuration only pay the price when things are
changed. Unfortunately this could still result in a lot of configuration
reloads for a single test class, like in the case for restconfig.

Pushing the idea a bit further, i had the idea of being more granular about
it. Doing something like this:

@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {
...
}

So instead of reloading the entire setup only a single store/layer/etc...
would have to be reloaded, or deleted, etc...

Anyways, just a rough idea at this point. Adopting helper annotations like
this would imply an upgrade of Junit to version 4. Which some tests are
actually already using afaik. There is also TestNG, which from a
customization point of view is much more flexible than Junit 4. Downside is
it is really a completely different framework, the upgrade would be much
less seamless and it has less built in support from our current maven and
eclipse tooling.

Are there tests using junit 4 in GeoServer?
Not against using Junit 4 or TestNG btw, but wondering how the current
opt-in "one time setup" mechanism can be replicated there.

I guess we might just not need the opt-in anymore if we switch to use
the annotations you're proposing though.

Eclipse wise there is a decent plugin for TestNG, thought I'm not too
happy about the way it works (maybe it's just a familiarity thing).
Don't know about the build server.

Regardless the switch to both JUnit 4 and TestNG would be a rather big
task. I can help some during the weekend if we're going there, but
it sounds like something that would require "all hands on board" to
be carried out (plus probably a GSIP to vote the switch).

Cheers
Andrea

--
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

I’ve thought a bit about trying to implement a copy-on-write catalog backend just for tests. The basic idea would be to have a datastore backend that uses one copy of the data directory that is never modified, and a directory containing modifications from this base. Any code that writes changes would receive file paths from the modifications directory, reads could happen directly based on the original. And resetting would just be a matter of throwing away the modifications directory (or even faster, making a new one and using that.) But I’m not sure how feasible it would be given the existing code base; we might be able to pull off a copy-on-read approach though.


David Winslow
OpenGeo - http://opengeo.org/

On Fri, Jun 15, 2012 at 5:06 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 15, 2012 at 4:38 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

(1) Keeping the same pattern as the existing test setup, we could simply
serialize out the new structure rather than the old structure. Should be
relatively straight forward and can probably use the facilities in
XStreamPersister to do so. An alternative would be to have a pre-canned data
directory and copy/unpack it when we have to do test setup.

The precanned one would be good for the tests that use the standard
mock data set, which is, I guess, the majority, so it sounds like a good idea.

However some setup only a handlful of types skipping the whole data dir
setup, so we need to be able to recognize if a test is setting up the base
data dir or not. Many tests add on top of it, but that’s something we can
handle after having setup the precanned one right?

(2) This is the interesting/tricky one. If tests that actually modify data
are going to utilize a one-time setup then what we need is way for a test
case to reload or refresh single layers at a time, returning both
configuration and data to original state. Or in cases where new layers are
added, remove them. Or if existing layers deleted, restore them. All of the
above I believe should really just amount to some utility classes on the
base test classes. However it does add a burden on the test writer since
they now have to worry about initialization/cleanup of state. Also there is
the issue of test order. The order of tests run in an IDE is not always the
same as running from maven and since state is maintained across test methods
i can see this becoming a mess in which failures occur with maven but not
from the ide.

Anyways, as an exercise I decided to locally port all the restconfig tests
to a one-time setup. As expected the results are significant. Currently on
trunk the full restconfig test run takes approximately 5 minutes on my local
machine. Utilizing the one time setup this drops down to about 1 minute 20
seconds.

So can we find a middle ground between using a one-time setup and making
test writers not have to worry too much about state between test methods?
One of the ideas i have is to use annotations to help with this. With Junit4
we essentially have the ability to define our own custom annotations, called
“rules”. So we could do something like this:

@Test
@PostReload
public void testFoo() {

}

The “PostReload” annotation would essentially tell the framework that the
test modifies configuration/data and after the test is run the configuration
must be reloaded. This would allow tests mixing test methods that are read
only with ones that modify configuration only pay the price when things are
changed. Unfortunately this could still result in a lot of configuration
reloads for a single test class, like in the case for restconfig.

Pushing the idea a bit further, i had the idea of being more granular about
it. Doing something like this:

@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {

}

So instead of reloading the entire setup only a single store/layer/etc…
would have to be reloaded, or deleted, etc…

Anyways, just a rough idea at this point. Adopting helper annotations like
this would imply an upgrade of Junit to version 4. Which some tests are
actually already using afaik. There is also TestNG, which from a
customization point of view is much more flexible than Junit 4. Downside is
it is really a completely different framework, the upgrade would be much
less seamless and it has less built in support from our current maven and
eclipse tooling.

Are there tests using junit 4 in GeoServer?
Not against using Junit 4 or TestNG btw, but wondering how the current
opt-in “one time setup” mechanism can be replicated there.

I guess we might just not need the opt-in anymore if we switch to use
the annotations you’re proposing though.

Eclipse wise there is a decent plugin for TestNG, thought I’m not too
happy about the way it works (maybe it’s just a familiarity thing).
Don’t know about the build server.

Regardless the switch to both JUnit 4 and TestNG would be a rather big
task. I can help some during the weekend if we’re going there, but
it sounds like something that would require “all hands on board” to
be carried out (plus probably a GSIP to vote the switch).

Cheers
Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Fri, Jun 15, 2012 at 3:06 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 15, 2012 at 4:38 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

(1) Keeping the same pattern as the existing test setup, we could simply
serialize out the new structure rather than the old structure. Should be
relatively straight forward and can probably use the facilities in
XStreamPersister to do so. An alternative would be to have a pre-canned data
directory and copy/unpack it when we have to do test setup.

The precanned one would be good for the tests that use the standard
mock data set, which is, I guess, the majority, so it sounds like a good idea.

However some setup only a handlful of types skipping the whole data dir
setup, so we need to be able to recognize if a test is setting up the base
data dir or not. Many tests add on top of it, but that’s something we can
handle after having setup the precanned one right?

When I did this locally i sort of ditched the pre-canned idea and just build up the resources programatically like we do now, just with the new structure. It was actually quite easy, just create a local catalog instance, attach a persister to it and start populating it.

(2) This is the interesting/tricky one. If tests that actually modify data
are going to utilize a one-time setup then what we need is way for a test
case to reload or refresh single layers at a time, returning both
configuration and data to original state. Or in cases where new layers are
added, remove them. Or if existing layers deleted, restore them. All of the
above I believe should really just amount to some utility classes on the
base test classes. However it does add a burden on the test writer since
they now have to worry about initialization/cleanup of state. Also there is
the issue of test order. The order of tests run in an IDE is not always the
same as running from maven and since state is maintained across test methods
i can see this becoming a mess in which failures occur with maven but not
from the ide.

Anyways, as an exercise I decided to locally port all the restconfig tests
to a one-time setup. As expected the results are significant. Currently on
trunk the full restconfig test run takes approximately 5 minutes on my local
machine. Utilizing the one time setup this drops down to about 1 minute 20
seconds.

So can we find a middle ground between using a one-time setup and making
test writers not have to worry too much about state between test methods?
One of the ideas i have is to use annotations to help with this. With Junit4
we essentially have the ability to define our own custom annotations, called
“rules”. So we could do something like this:

@Test
@PostReload
public void testFoo() {

}

The “PostReload” annotation would essentially tell the framework that the
test modifies configuration/data and after the test is run the configuration
must be reloaded. This would allow tests mixing test methods that are read
only with ones that modify configuration only pay the price when things are
changed. Unfortunately this could still result in a lot of configuration
reloads for a single test class, like in the case for restconfig.

Pushing the idea a bit further, i had the idea of being more granular about
it. Doing something like this:

@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {

}

So instead of reloading the entire setup only a single store/layer/etc…
would have to be reloaded, or deleted, etc…

Anyways, just a rough idea at this point. Adopting helper annotations like
this would imply an upgrade of Junit to version 4. Which some tests are
actually already using afaik. There is also TestNG, which from a
customization point of view is much more flexible than Junit 4. Downside is
it is really a completely different framework, the upgrade would be much
less seamless and it has less built in support from our current maven and
eclipse tooling.

Are there tests using junit 4 in GeoServer?

Maybe not, i was probably thinking of geotools.

Not against using Junit 4 or TestNG btw, but wondering how the current
opt-in “one time setup” mechanism can be replicated there.

I guess we might just not need the opt-in anymore if we switch to use
the annotations you’re proposing though.

Right, this is more of what I was thinking. Basically I was thinking that you would have to explicitly “opt-out” of the one time setup and by default have it used. The idea with the annotations would give you control and if need be trigger a reload before a test. Or refresh particular resources after a test, etc…

Eclipse wise there is a decent plugin for TestNG, thought I’m not too
happy about the way it works (maybe it’s just a familiarity thing).
Don’t know about the build server.

Yeah, i haven’t tried the plugin at all. TestNG does seem to offer a lot more options though. Of particular interest to me are test groups and parallelism.

Regardless the switch to both JUnit 4 and TestNG would be a rather big
task. I can help some during the weekend if we’re going there, but
it sounds like something that would require “all hands on board” to
be carried out (plus probably a GSIP to vote the switch).

A big task and proposal worthy indeed. At this point just something i am doing some initial experimentation on brought about by the fact that doing full test runs are pretty painful.

As for how to migrate I am sort of thinking an incremental approach here. Basically the idea would be come up with a new base test class that tests would extend that implement the new testing setup. And port modules over it to one-by-one.

Cheers
Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

On Fri, Jun 15, 2012 at 8:18 AM, David Winslow <dwinslow@anonymised.com> wrote:

I’ve thought a bit about trying to implement a copy-on-write catalog backend just for tests. The basic idea would be to have a datastore backend that uses one copy of the data directory that is never modified, and a directory containing modifications from this base. Any code that writes changes would receive file paths from the modifications directory, reads could happen directly based on the original. And resetting would just be a matter of throwing away the modifications directory (or even faster, making a new one and using that.) But I’m not sure how feasible it would be given the existing code base; we might be able to pull off a copy-on-read approach though.

Interesting idea. I actually kind of had a similar one but at a bit of a higher level, and that is the idea of a kind of a catalog wrapper that records every operation against the catalog and for each operation maintain the inverse of it on a stack. After a test run the inverse operations would be played back essentially rolling back any changes made.


David Winslow
OpenGeo - http://opengeo.org/

On Fri, Jun 15, 2012 at 5:06 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 15, 2012 at 4:38 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

(1) Keeping the same pattern as the existing test setup, we could simply
serialize out the new structure rather than the old structure. Should be
relatively straight forward and can probably use the facilities in
XStreamPersister to do so. An alternative would be to have a pre-canned data
directory and copy/unpack it when we have to do test setup.

The precanned one would be good for the tests that use the standard
mock data set, which is, I guess, the majority, so it sounds like a good idea.

However some setup only a handlful of types skipping the whole data dir
setup, so we need to be able to recognize if a test is setting up the base
data dir or not. Many tests add on top of it, but that’s something we can
handle after having setup the precanned one right?

(2) This is the interesting/tricky one. If tests that actually modify data
are going to utilize a one-time setup then what we need is way for a test
case to reload or refresh single layers at a time, returning both
configuration and data to original state. Or in cases where new layers are
added, remove them. Or if existing layers deleted, restore them. All of the
above I believe should really just amount to some utility classes on the
base test classes. However it does add a burden on the test writer since
they now have to worry about initialization/cleanup of state. Also there is
the issue of test order. The order of tests run in an IDE is not always the
same as running from maven and since state is maintained across test methods
i can see this becoming a mess in which failures occur with maven but not
from the ide.

Anyways, as an exercise I decided to locally port all the restconfig tests
to a one-time setup. As expected the results are significant. Currently on
trunk the full restconfig test run takes approximately 5 minutes on my local
machine. Utilizing the one time setup this drops down to about 1 minute 20
seconds.

So can we find a middle ground between using a one-time setup and making
test writers not have to worry too much about state between test methods?
One of the ideas i have is to use annotations to help with this. With Junit4
we essentially have the ability to define our own custom annotations, called
“rules”. So we could do something like this:

@Test
@PostReload
public void testFoo() {

}

The “PostReload” annotation would essentially tell the framework that the
test modifies configuration/data and after the test is run the configuration
must be reloaded. This would allow tests mixing test methods that are read
only with ones that modify configuration only pay the price when things are
changed. Unfortunately this could still result in a lot of configuration
reloads for a single test class, like in the case for restconfig.

Pushing the idea a bit further, i had the idea of being more granular about
it. Doing something like this:

@Test
@PostReload(store=sf)
@PostRemove(layer=sf:newLayer)
public void testFoo() {

}

So instead of reloading the entire setup only a single store/layer/etc…
would have to be reloaded, or deleted, etc…

Anyways, just a rough idea at this point. Adopting helper annotations like
this would imply an upgrade of Junit to version 4. Which some tests are
actually already using afaik. There is also TestNG, which from a
customization point of view is much more flexible than Junit 4. Downside is
it is really a completely different framework, the upgrade would be much
less seamless and it has less built in support from our current maven and
eclipse tooling.

Are there tests using junit 4 in GeoServer?
Not against using Junit 4 or TestNG btw, but wondering how the current
opt-in “one time setup” mechanism can be replicated there.

I guess we might just not need the opt-in anymore if we switch to use
the annotations you’re proposing though.

Eclipse wise there is a decent plugin for TestNG, thought I’m not too
happy about the way it works (maybe it’s just a familiarity thing).
Don’t know about the build server.

Regardless the switch to both JUnit 4 and TestNG would be a rather big
task. I can help some during the weekend if we’re going there, but
it sounds like something that would require “all hands on board” to
be carried out (plus probably a GSIP to vote the switch).

Cheers
Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

On Fri, Jun 15, 2012 at 5:06 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

However some setup only a handlful of types skipping the whole data dir
setup, so we need to be able to recognize if a test is setting up the base
data dir or not. Many tests add on top of it, but that's something we can
handle after having setup the precanned one right?

When I did this locally i sort of ditched the pre-canned idea and just build
up the resources programatically like we do now, just with the new
structure. It was actually quite easy, just create a local catalog instance,
attach a persister to it and start populating it.

Works for me

Eclipse wise there is a decent plugin for TestNG, thought I'm not too
happy about the way it works (maybe it's just a familiarity thing).
Don't know about the build server.

Yeah, i haven't tried the plugin at all. TestNG does seem to offer a lot
more options though. Of particular interest to me are test groups
and parallelism.

Regardless the switch to both JUnit 4 and TestNG would be a rather big
task. I can help some during the weekend if we're going there, but
it sounds like something that would require "all hands on board" to
be carried out (plus probably a GSIP to vote the switch).

A big task and proposal worthy indeed. At this point just something i am
doing some initial experimentation on brought about by the fact that doing
full test runs are pretty painful.

As for how to migrate I am sort of thinking an incremental approach here.
Basically the idea would be come up with a new base test class that tests
would extend that implement the new testing setup. And port modules over it
to one-by-one.

I'm on board with this, but wouldn't this prevent a switch to TestNG or Junit 4?
That is, how does one have a base class that works both for JUnit 3
and Junit 4/TestNG,
leaving the existin tests untouched?

Another possibility could be to have our JUnit 3 base class process
annotations and
handle them accordingly (ok, not nice, but probably less work)

Cheers
Andrea

--
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

On Fri, Jun 15, 2012 at 11:13 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

I’m on board with this, but wouldn’t this prevent a switch to TestNG or Junit 4?
That is, how does one have a base class that works both for JUnit 3
and Junit 4/TestNG,
leaving the existin tests untouched?

Another possibility could be to have our JUnit 3 base class process
annotations and
handle them accordingly (ok, not nice, but probably less work)

One approach might be to factor out the “mock” geoserver setup to a class that isn’t involved in the test suite hierarchy. The Wicket WicketTester class is a good example of doing this - you don’t inherit from “BaseWicketTestSupport,” you just add a WicketTester field (or even a local variable.) Similarly, we could have a GeoServerTester that encapsulates the mock servlet container and client emulation API from the JUnit ones. I think we could make that migration fairly smooth by having GeoServerTestSupport just delegate appropriate calls to a GeoServerTester managed by its setup and teardown methods.


David Winslow
OpenGeo - http://opengeo.org/

On Fri, Jun 15, 2012 at 9:13 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 15, 2012 at 5:06 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

However some setup only a handlful of types skipping the whole data dir
setup, so we need to be able to recognize if a test is setting up the base
data dir or not. Many tests add on top of it, but that’s something we can
handle after having setup the precanned one right?

When I did this locally i sort of ditched the pre-canned idea and just build
up the resources programatically like we do now, just with the new
structure. It was actually quite easy, just create a local catalog instance,
attach a persister to it and start populating it.

Works for me

Eclipse wise there is a decent plugin for TestNG, thought I’m not too
happy about the way it works (maybe it’s just a familiarity thing).
Don’t know about the build server.

Yeah, i haven’t tried the plugin at all. TestNG does seem to offer a lot
more options though. Of particular interest to me are test groups
and parallelism.

Regardless the switch to both JUnit 4 and TestNG would be a rather big
task. I can help some during the weekend if we’re going there, but
it sounds like something that would require “all hands on board” to
be carried out (plus probably a GSIP to vote the switch).

A big task and proposal worthy indeed. At this point just something i am
doing some initial experimentation on brought about by the fact that doing
full test runs are pretty painful.

As for how to migrate I am sort of thinking an incremental approach here.
Basically the idea would be come up with a new base test class that tests
would extend that implement the new testing setup. And port modules over it
to one-by-one.

I’m on board with this, but wouldn’t this prevent a switch to TestNG or Junit 4?
That is, how does one have a base class that works both for JUnit 3
and Junit 4/TestNG,
leaving the existin tests untouched?

The idea is that the existing base would remain a junit3 style, and the new one would be based on junit4 (if we go that way). So basically first step would be to upgrade codebase to junit 4 which shouldn’t affect anything, the old style tests will still run without issue.

Then when someone goes to port a test case over the workflow is basically:

  1. Switch to the new base class (designed to be a pretty seamless drop in replacement)
  2. Annotate all test methods

Note that TestNG actually would make this easier in this regard, it allows you to annotate the class itself with a “@Test” annotation and when that happens it basically treats every public method as a test case.

Another possibility could be to have our JUnit 3 base class process
annotations and
handle them accordingly (ok, not nice, but probably less work)

Cheers
Andrea


Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.