[Geoserver-devel] Some ideas on how to allow unit testing with other datastores

Hi,
lately I've been having quite a bit of troubles in the WFS versioning
module. The reason is simple, I cannot unit test it because we don't
have a "versioning property datastore", that is, a datastore that
makes no assumptions on external software installed on the machine.

Soo, it seems I need some way to test using the postgis versioning
datastore. This means I need:
* a way to check a certain postgis datastore is available on the
   machine (so that I can skip the test if it's not there)
* a way to create a data directory using that datastore
* a way to run setup and cleanup scripts so that I can trust
   the initial state of the db

The gt2 do something similar in jdbc datastores by making use
of a property file. They then build a connection by hand and run
whatever sql is needed directly from code.

Now, the current testing base class (GeoServerTestSupport)
provides testing convinience methods I need to keep
(methods to execute a post call and parse the results into
a DOM, for example), and a direct reference to MockData
class, that I want to remove.

So what I propose is to move MockData usage in a subclass,
GeoServerMockTestSupport, and leave the base class emtpy with
regards to how the data dir is setup. The base class would
become abstract with a template method to gather the actual
location of the data dir.
This way I can create another subclass that takes care of using
a predefined data dir instead and do the above.

To recap, I make a new GeoServetTestSupport with an abstract
method String getDataDirLocation(). I create a GeoServerMockTestSupport
that uses MockData and instantiates it in getDataDirLocation(), and
this class becomes the new base class of all existing unit tests.
Finally, I'll create in the WFSV module a new subclass allowing
the usage of the postgis versioning data store.

The idea is to store a minimal but working
data dir as a test resource, parse catalog.xml (with XMLConfigReader)
in order to gather the connection params,
test the db is there, run programmatically an sql script, and
then start up the mock geoserver for testing (the sql
script should take care of dropping the tables before re-creating
them, so that we have a single sql script).
After that, we set the test data dir path into the servlet
context and we start up the unit testing.

The data dir could be used as is, without being copied, since
no one of the tests really need to modify it (the wildest
tests may need to alter the in memory representation, but not
to actually change the stored config).

What do you think? Seems simple enough. If we want a more
abstract superclass could be used to allow doing the same
kind of testing against Oracle or Postgis (should we ever
stumble into a datastore specific issue we don't feel
comfortable unit testing in gt2 only).

Cheers
Andrea

ahhh... the good old testing issues..
first, Andrea, I'd say go for it, as it comes from a real need and sure you
need it asap.

That said, I'm a bit worried about our test mess up.
In an ideal world, I'd like to have _unit_ tests separated from _integration_
tests, and why not to also have a set of _system_ tests and guess we could
take CITE as our acceptance tests, though it could obviously be augmented.
There are lots of test types a project may be exposed to, yet I recognize
being strict on doing all of them would be quite much, but I guess the
unit/integration/functional/performance/acceptance is a nice separation.

That is to say I guess our test suite mixes them all and does none well
enough, like in we shouldn't need a real datastore for _unit_ testing (at
this stage you only assert each unit does things right). The "mocked up"
properties one could be used for integration (where you test the classes you
just unit testes collaborate well together), and then we do need real
datastores for functional/system tests (where you ensure the system does the
right thing)

As for unit testing, I'm lately using EasyMock <http://www.easymock.org/&gt;,
which provides a nice way to mock up the collaborators a class need among
allowing to ensure those collaborators were properly used by the class under
test. This provides a nice isolation path in order to achieve true unit
testing.

As for performance/load/scalability JMeter seems like good candidate.

well, testing is a very vast topic and I'm far from being an expert, just that
I was reading a bit about it and implementing these separation of concerns on
another project and am enjoying it and would like to see it on geoserver too.

what do you think? is it something we should plan on for the mid term?

cheers,

Gabriel

On Tuesday 18 December 2007 06:47:16 pm Andrea Aime wrote:

Hi,
lately I've been having quite a bit of troubles in the WFS versioning
module. The reason is simple, I cannot unit test it because we don't
have a "versioning property datastore", that is, a datastore that
makes no assumptions on external software installed on the machine.

Soo, it seems I need some way to test using the postgis versioning
datastore. This means I need:
* a way to check a certain postgis datastore is available on the
   machine (so that I can skip the test if it's not there)
* a way to create a data directory using that datastore
* a way to run setup and cleanup scripts so that I can trust
   the initial state of the db

The gt2 do something similar in jdbc datastores by making use
of a property file. They then build a connection by hand and run
whatever sql is needed directly from code.

Now, the current testing base class (GeoServerTestSupport)
provides testing convinience methods I need to keep
(methods to execute a post call and parse the results into
a DOM, for example), and a direct reference to MockData
class, that I want to remove.

So what I propose is to move MockData usage in a subclass,
GeoServerMockTestSupport, and leave the base class emtpy with
regards to how the data dir is setup. The base class would
become abstract with a template method to gather the actual
location of the data dir.
This way I can create another subclass that takes care of using
a predefined data dir instead and do the above.

To recap, I make a new GeoServetTestSupport with an abstract
method String getDataDirLocation(). I create a GeoServerMockTestSupport
that uses MockData and instantiates it in getDataDirLocation(), and
this class becomes the new base class of all existing unit tests.
Finally, I'll create in the WFSV module a new subclass allowing
the usage of the postgis versioning data store.

The idea is to store a minimal but working
data dir as a test resource, parse catalog.xml (with XMLConfigReader)
in order to gather the connection params,
test the db is there, run programmatically an sql script, and
then start up the mock geoserver for testing (the sql
script should take care of dropping the tables before re-creating
them, so that we have a single sql script).
After that, we set the test data dir path into the servlet
context and we start up the unit testing.

The data dir could be used as is, without being copied, since
no one of the tests really need to modify it (the wildest
tests may need to alter the in memory representation, but not
to actually change the stored config).

What do you think? Seems simple enough. If we want a more
abstract superclass could be used to allow doing the same
kind of testing against Oracle or Postgis (should we ever
stumble into a datastore specific issue we don't feel
comfortable unit testing in gt2 only).

Cheers
Andrea

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplac
e _______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4045,476809a8159755219720167!

+1 from me on the idea Andrea. One question though. How will these unit
tests be turned on and off. I guess a simple check for if the defined
data directory exists should do it?

Gabriel Roldán wrote:

ahhh... the good old testing issues..
first, Andrea, I'd say go for it, as it comes from a real need and sure you
need it asap.

That said, I'm a bit worried about our test mess up.
In an ideal world, I'd like to have _unit_ tests separated from _integration_
tests, and why not to also have a set of _system_ tests and guess we could
take CITE as our acceptance tests, though it could obviously be augmented.
There are lots of test types a project may be exposed to, yet I recognize
being strict on doing all of them would be quite much, but I guess the
unit/integration/functional/performance/acceptance is a nice separation.

That is to say I guess our test suite mixes them all and does none well
enough, like in we shouldn't need a real datastore for _unit_ testing (at
this stage you only assert each unit does things right). The "mocked up"
properties one could be used for integration (where you test the classes you
just unit testes collaborate well together), and then we do need real
datastores for functional/system tests (where you ensure the system does the
right thing)

As for unit testing, I'm lately using EasyMock <http://www.easymock.org/&gt;,
which provides a nice way to mock up the collaborators a class need among
allowing to ensure those collaborators were properly used by the class under
test. This provides a nice isolation path in order to achieve true unit
testing.

As for performance/load/scalability JMeter seems like good candidate.

well, testing is a very vast topic and I'm far from being an expert, just that
I was reading a bit about it and implementing these separation of concerns on
another project and am enjoying it and would like to see it on geoserver too.

what do you think? is it something we should plan on for the mid term?

cheers,

Gabriel

On Tuesday 18 December 2007 06:47:16 pm Andrea Aime wrote:

Hi,
lately I've been having quite a bit of troubles in the WFS versioning
module. The reason is simple, I cannot unit test it because we don't
have a "versioning property datastore", that is, a datastore that
makes no assumptions on external software installed on the machine.

Soo, it seems I need some way to test using the postgis versioning
datastore. This means I need:
* a way to check a certain postgis datastore is available on the
   machine (so that I can skip the test if it's not there)
* a way to create a data directory using that datastore
* a way to run setup and cleanup scripts so that I can trust
   the initial state of the db

The gt2 do something similar in jdbc datastores by making use
of a property file. They then build a connection by hand and run
whatever sql is needed directly from code.

Now, the current testing base class (GeoServerTestSupport)
provides testing convinience methods I need to keep
(methods to execute a post call and parse the results into
a DOM, for example), and a direct reference to MockData
class, that I want to remove.

So what I propose is to move MockData usage in a subclass,
GeoServerMockTestSupport, and leave the base class emtpy with
regards to how the data dir is setup. The base class would
become abstract with a template method to gather the actual
location of the data dir.
This way I can create another subclass that takes care of using
a predefined data dir instead and do the above.

To recap, I make a new GeoServetTestSupport with an abstract
method String getDataDirLocation(). I create a GeoServerMockTestSupport
that uses MockData and instantiates it in getDataDirLocation(), and
this class becomes the new base class of all existing unit tests.
Finally, I'll create in the WFSV module a new subclass allowing
the usage of the postgis versioning data store.

The idea is to store a minimal but working
data dir as a test resource, parse catalog.xml (with XMLConfigReader)
in order to gather the connection params,
test the db is there, run programmatically an sql script, and
then start up the mock geoserver for testing (the sql
script should take care of dropping the tables before re-creating
them, so that we have a single sql script).
After that, we set the test data dir path into the servlet
context and we start up the unit testing.

The data dir could be used as is, without being copied, since
no one of the tests really need to modify it (the wildest
tests may need to alter the in memory representation, but not
to actually change the stored config).

What do you think? Seems simple enough. If we want a more
abstract superclass could be used to allow doing the same
kind of testing against Oracle or Postgis (should we ever
stumble into a datastore specific issue we don't feel
comfortable unit testing in gt2 only).

Cheers
Andrea

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplac
e _______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,47681a38197031628642973!

--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org

You bring up a great point Gabriel, in that 95% of ours tests
(mock+cite) start strictly integration tests. However, the facility to
unit test is still there. On each test the entire container is "mocked"
up. There is nothing stopping us from grabbing specific beans from the
spring container and executing unit tests against them.

Perhaps we should come up with another base class called
"GeoServerUnitTestSupport", which would not provide the same methods to
execute mock requests, but provide components from the container for
unit testing. We could also play off Andreas original idea of factoring
out the "DataDirectory". Unit tests could run "empty", making them easy
to tear down and set up. However I suppose that for unit testing some
data will need to be available... not sure.

As for performance testing, jmeter is a good candidate... but running it
in an offline mode proved to be a bit difficult. Although Andrea seemed
to get it to go. One thing I have been wanting to look at for a while is
JUnitPerf.

One thing I really want to see is a performance testing server which
could run and publish reports nicely. Lets chat about this at todays IRC
meeting.

-Justin

Gabriel Roldán wrote:

ahhh... the good old testing issues..
first, Andrea, I'd say go for it, as it comes from a real need and sure you
need it asap.

That said, I'm a bit worried about our test mess up.
In an ideal world, I'd like to have _unit_ tests separated from _integration_
tests, and why not to also have a set of _system_ tests and guess we could
take CITE as our acceptance tests, though it could obviously be augmented.
There are lots of test types a project may be exposed to, yet I recognize
being strict on doing all of them would be quite much, but I guess the
unit/integration/functional/performance/acceptance is a nice separation.

That is to say I guess our test suite mixes them all and does none well
enough, like in we shouldn't need a real datastore for _unit_ testing (at
this stage you only assert each unit does things right). The "mocked up"
properties one could be used for integration (where you test the classes you
just unit testes collaborate well together), and then we do need real
datastores for functional/system tests (where you ensure the system does the
right thing)

As for unit testing, I'm lately using EasyMock <http://www.easymock.org/&gt;,
which provides a nice way to mock up the collaborators a class need among
allowing to ensure those collaborators were properly used by the class under
test. This provides a nice isolation path in order to achieve true unit
testing.

As for performance/load/scalability JMeter seems like good candidate.

well, testing is a very vast topic and I'm far from being an expert, just that
I was reading a bit about it and implementing these separation of concerns on
another project and am enjoying it and would like to see it on geoserver too.

what do you think? is it something we should plan on for the mid term?

cheers,

Gabriel

On Tuesday 18 December 2007 06:47:16 pm Andrea Aime wrote:

Hi,
lately I've been having quite a bit of troubles in the WFS versioning
module. The reason is simple, I cannot unit test it because we don't
have a "versioning property datastore", that is, a datastore that
makes no assumptions on external software installed on the machine.

Soo, it seems I need some way to test using the postgis versioning
datastore. This means I need:
* a way to check a certain postgis datastore is available on the
   machine (so that I can skip the test if it's not there)
* a way to create a data directory using that datastore
* a way to run setup and cleanup scripts so that I can trust
   the initial state of the db

The gt2 do something similar in jdbc datastores by making use
of a property file. They then build a connection by hand and run
whatever sql is needed directly from code.

Now, the current testing base class (GeoServerTestSupport)
provides testing convinience methods I need to keep
(methods to execute a post call and parse the results into
a DOM, for example), and a direct reference to MockData
class, that I want to remove.

So what I propose is to move MockData usage in a subclass,
GeoServerMockTestSupport, and leave the base class emtpy with
regards to how the data dir is setup. The base class would
become abstract with a template method to gather the actual
location of the data dir.
This way I can create another subclass that takes care of using
a predefined data dir instead and do the above.

To recap, I make a new GeoServetTestSupport with an abstract
method String getDataDirLocation(). I create a GeoServerMockTestSupport
that uses MockData and instantiates it in getDataDirLocation(), and
this class becomes the new base class of all existing unit tests.
Finally, I'll create in the WFSV module a new subclass allowing
the usage of the postgis versioning data store.

The idea is to store a minimal but working
data dir as a test resource, parse catalog.xml (with XMLConfigReader)
in order to gather the connection params,
test the db is there, run programmatically an sql script, and
then start up the mock geoserver for testing (the sql
script should take care of dropping the tables before re-creating
them, so that we have a single sql script).
After that, we set the test data dir path into the servlet
context and we start up the unit testing.

The data dir could be used as is, without being copied, since
no one of the tests really need to modify it (the wildest
tests may need to alter the in memory representation, but not
to actually change the stored config).

What do you think? Seems simple enough. If we want a more
abstract superclass could be used to allow doing the same
kind of testing against Oracle or Postgis (should we ever
stumble into a datastore specific issue we don't feel
comfortable unit testing in gt2 only).

Cheers
Andrea

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplac
e _______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,47681a38197031628642973!

--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org

Justin Deoliveira ha scritto:

+1 from me on the idea Andrea. One question though. How will these unit
tests be turned on and off. I guess a simple check for if the defined
data directory exists should do it?

The idea is that the data directory is a test resource, so it
comes along with the source code using it. XMLConfigReader is
used to read catalog.xml, gather the connection params. If
the connection to the db is successfull, then the test is
run, otherwise not. This check is done once in the base class
and stored in a static field so that it does not slow down
significantly testing.

Cheers
Andrea