[Geoserver-devel] JUnit3, JUnit4 and TestNG comparison (for one time setup)JUnit3, JUnit4 and TestNG comparison (for one time setup)

Hi,
I've made a comparison between the approaches one can take to
have a one time setup/teardown in order to avoid going thru
the initialization of geoserver over and over and over
during tests (at least for read only ones).

The extra requirement is that the one time setup has
to allow calling non static methods, since in GeoServetTestSupport
we have 3 template methods that subclasses can alter
in order to drive how the setup is done.

This project:
http://www.nabble.com/file/p16609696/onetimesetup.zip onetimesetup.zip
shows how one time setup/teardown
can be achieved using the three testing frameworks.

The first results is that junit4 and junit3 tests can
coexist in the same project, but testng not. Once
you add testng to the classpath, surefire will use only
that one to run the tests, and a class cast exception
occurrs trying to run the junit3 tests (probably due
to the suite() method, not sure).

Anyways, besides this, here is a rough summary.

JUnit3
---------------------------------------------------
The base class, OneTimeSetupJunit3, sets the scene
for one time setup. Class extending it can specify
the following methods:
public void oneTimeSetUp()
public void oneTimeTearDown()
protected void setUpInternal()
protected void tearDownInternal()

If only those are specified, the order of execution
*for each test method* is:
oneTimeSetup
setUpInternal
tearDownInternal
oneTimeTearDown
(that is, the one time methods, if present, behave
like normal setup/teardown would).

If the following method is added:
public static Test suite() {
  return new OneTimeTestSetup(new OneTimeJunit3Test());
}

then the one time setup and tear down methods are run
only before running all tests, and after running them
(so they become true one time setup/teardown).

All fine? No. When you add that suite() method,
Eclipse will run all the tests even if you ask it
to run just once. So if you need to debug just one
method, that suite() method has to be commented
out, and uncommented before committing. How inconvinient.

Junit4
-----------------------------------------------------
The base class, OneTimeSetupJunit4, tries to allow
for the same approach as the OneTimeSetupJunit4.
Unfortunately I could not manage to allow for
an ovveridable OneTimeTearDown.

Yet, for our specific case, it's not really needed,
since all template methods we need to call are
in the setup phase, so we could get away marking
the only one OneTimeTearDown method we need
as @AfterClass

TestNG
----------------------------------------------------
Look ma, no base class. The annotations here are placed
on standard non static methods, so no machinery
is really needed.
To run the TestNG tests in Eclipse a plugin is
needed, that you can download from here:
http://testng.org/doc/download.html

My take
----------------------------------------------------
Hum, the junit3 approach is a little annyoing but
works, and does not require us to learn anything new.
The TestNG one is the one that looks better thought,
no need to fight against the framework.

So, what shall we do? Blue pill, junit3, red pill,
juni4, green pill, TestNG.
What are you going to swallow?

Please vote
Cheers
Andrea

--
View this message in context: http://www.nabble.com/JUnit3%2C-JUnit4-and-TestNG-comparison-(for-one-time-setup)JUnit3%2C-JUnit4-and-TestNG-comparison-(for-one-time-setup)-tp16609696p16609696.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

Hi Andrea,

the zip file reffered seems to no longer exist. Can you send it again another
way?

cheers,

Gabriel
On Thursday 10 April 2008 05:58:55 pm aaime wrote:

Hi,
I've made a comparison between the approaches one can take to
have a one time setup/teardown in order to avoid going thru
the initialization of geoserver over and over and over
during tests (at least for read only ones).

The extra requirement is that the one time setup has
to allow calling non static methods, since in GeoServetTestSupport
we have 3 template methods that subclasses can alter
in order to drive how the setup is done.

This project:
http://www.nabble.com/file/p16609696/onetimesetup.zip onetimesetup.zip
shows how one time setup/teardown
can be achieved using the three testing frameworks.

The first results is that junit4 and junit3 tests can
coexist in the same project, but testng not. Once
you add testng to the classpath, surefire will use only
that one to run the tests, and a class cast exception
occurrs trying to run the junit3 tests (probably due
to the suite() method, not sure).

Anyways, besides this, here is a rough summary.

JUnit3
---------------------------------------------------
The base class, OneTimeSetupJunit3, sets the scene
for one time setup. Class extending it can specify
the following methods:
public void oneTimeSetUp()
public void oneTimeTearDown()
protected void setUpInternal()
protected void tearDownInternal()

If only those are specified, the order of execution
*for each test method* is:
oneTimeSetup
setUpInternal
tearDownInternal
oneTimeTearDown
(that is, the one time methods, if present, behave
like normal setup/teardown would).

If the following method is added:
public static Test suite() {
  return new OneTimeTestSetup(new OneTimeJunit3Test());
}

then the one time setup and tear down methods are run
only before running all tests, and after running them
(so they become true one time setup/teardown).

All fine? No. When you add that suite() method,
Eclipse will run all the tests even if you ask it
to run just once. So if you need to debug just one
method, that suite() method has to be commented
out, and uncommented before committing. How inconvinient.

Junit4
-----------------------------------------------------
The base class, OneTimeSetupJunit4, tries to allow
for the same approach as the OneTimeSetupJunit4.
Unfortunately I could not manage to allow for
an ovveridable OneTimeTearDown.

Yet, for our specific case, it's not really needed,
since all template methods we need to call are
in the setup phase, so we could get away marking
the only one OneTimeTearDown method we need
as @AfterClass

TestNG
----------------------------------------------------
Look ma, no base class. The annotations here are placed
on standard non static methods, so no machinery
is really needed.
To run the TestNG tests in Eclipse a plugin is
needed, that you can download from here:
http://testng.org/doc/download.html

My take
----------------------------------------------------
Hum, the junit3 approach is a little annyoing but
works, and does not require us to learn anything new.
The TestNG one is the one that looks better thought,
no need to fight against the framework.

So, what shall we do? Blue pill, junit3, red pill,
juni4, green pill, TestNG.
What are you going to swallow?

Please vote
Cheers
Andrea

Gabriel Roldán ha scritto:

Hi Andrea,

the zip file reffered seems to no longer exist. Can you send it again another way?

Ah, damn, it's just that I could not attach it
to the mail...

Anyways, here we go:
http://geo.openplans.org/~aaime/onetimesetup.zip

Cheers
Andrea

Well you know me... i am a sucker for pretty looking code. But testng not playing nice with surefire seems like a bit of a blocker. So that leaves us with Junit3 or Junit4.

Question: will we be able to do the setup in the base class alone? Or is this something that individual tests will need to know about?

aaime wrote:

Hi,
I've made a comparison between the approaches one can take to
have a one time setup/teardown in order to avoid going thru
the initialization of geoserver over and over and over
during tests (at least for read only ones).

The extra requirement is that the one time setup has
to allow calling non static methods, since in GeoServetTestSupport
we have 3 template methods that subclasses can alter
in order to drive how the setup is done.

This project:
http://www.nabble.com/file/p16609696/onetimesetup.zip onetimesetup.zip shows how one time setup/teardown
can be achieved using the three testing frameworks.

The first results is that junit4 and junit3 tests can
coexist in the same project, but testng not. Once
you add testng to the classpath, surefire will use only
that one to run the tests, and a class cast exception
occurrs trying to run the junit3 tests (probably due
to the suite() method, not sure).

Anyways, besides this, here is a rough summary.

JUnit3
---------------------------------------------------
The base class, OneTimeSetupJunit3, sets the scene
for one time setup. Class extending it can specify
the following methods:
public void oneTimeSetUp()
public void oneTimeTearDown()
protected void setUpInternal()
protected void tearDownInternal()

If only those are specified, the order of execution
*for each test method* is:
oneTimeSetup
setUpInternal
tearDownInternal
oneTimeTearDown
(that is, the one time methods, if present, behave
like normal setup/teardown would).

If the following method is added:
public static Test suite() {
  return new OneTimeTestSetup(new OneTimeJunit3Test());
}

then the one time setup and tear down methods are run
only before running all tests, and after running them
(so they become true one time setup/teardown).

All fine? No. When you add that suite() method,
Eclipse will run all the tests even if you ask it
to run just once. So if you need to debug just one
method, that suite() method has to be commented
out, and uncommented before committing. How inconvinient.

Junit4
-----------------------------------------------------
The base class, OneTimeSetupJunit4, tries to allow
for the same approach as the OneTimeSetupJunit4.
Unfortunately I could not manage to allow for
an ovveridable OneTimeTearDown.

Yet, for our specific case, it's not really needed,
since all template methods we need to call are
in the setup phase, so we could get away marking
the only one OneTimeTearDown method we need
as @AfterClass

TestNG
----------------------------------------------------
Look ma, no base class. The annotations here are placed
on standard non static methods, so no machinery
is really needed.
To run the TestNG tests in Eclipse a plugin is
needed, that you can download from here:
http://testng.org/doc/download.html

My take
----------------------------------------------------
Hum, the junit3 approach is a little annyoing but
works, and does not require us to learn anything new.
The TestNG one is the one that looks better thought,
no need to fight against the framework.

So, what shall we do? Blue pill, junit3, red pill,
juni4, green pill, TestNG.
What are you going to swallow?

Please vote
Cheers
Andrea

--
Justin Deoliveira
The Open Planning Project
jdeolive@anonymised.com

Justin Deoliveira ha scritto:

Well you know me... i am a sucker for pretty looking code. But testng not playing nice with surefire seems like a bit of a blocker. So that leaves us with Junit3 or Junit4.

Yeah... well, the thing is, if you choose the right testng it works
with surefire, but it tends to lag behind the released versions of
testng (works with 5.1, but latest is 5.5 or something like that).

And of course there is the "mixing tests" issue, thought I may
be due to the fact that one of my juni3 classes defines a suite() method.

Question: will we be able to do the setup in the base class alone? Or is this something that individual tests will need to know about?

No, unfortunately it's not possible. With junit4 I did not manage to
get it fully working (one time tear down I cannot define, so the
subclass would have to manually invoke it), with junit3 you have
to define that suite() method and the comment it out each time
you have to run a single test (otherwise due to the suite() method
being there, you end up running them all anyways).

So yeah, so far an unsatisfactory situation. I'm hoping Gabriel,
by the power of the unit testing book he ate, will find a better
solution.

Cheers
Andrea

Andrea Aime ha scritto:

No, unfortunately it's not possible. With junit4 I did not manage to
get it fully working (one time tear down I cannot define,

More precisely, I cannot define a one time tear down method that
calls non static methods (template methods in the subclasses).
Yet, as far as we're concerned today, that is not needed,
the template methods are called only by setup.

So the answer is yes, with the current situation, it's doable
with junit4. Of course the subclass has to say it's ok to keep
the test fixture around for multiple tests, there is a method
the subclass has to override and return "true" to say it's ok.

Cheers
Andrea

On Thursday 10 April 2008 11:54:30 pm Andrea Aime wrote:

Justin Deoliveira ha scritto:
> Well you know me... i am a sucker for pretty looking code. But testng
> not playing nice with surefire seems like a bit of a blocker. So that
> leaves us with Junit3 or Junit4.

Yeah... well, the thing is, if you choose the right testng it works
with surefire, but it tends to lag behind the released versions of
testng (works with 5.1, but latest is 5.5 or something like that).

And of course there is the "mixing tests" issue, thought I may
be due to the fact that one of my juni3 classes defines a suite() method.

> Question: will we be able to do the setup in the base class alone? Or is
> this something that individual tests will need to know about?

No, unfortunately it's not possible. With junit4 I did not manage to
get it fully working (one time tear down I cannot define, so the
subclass would have to manually invoke it), with junit3 you have
to define that suite() method and the comment it out each time
you have to run a single test (otherwise due to the suite() method
being there, you end up running them all anyways).

here is where I'm getting lost. I can swear I can run a single test case from
ArcSdeDataStoreTest inside eclipse, even if it uses the junit3 "static
suite() returns wrapper" approach, since suite() is not called at all when
running a single test from inside eclipse, I just need to check in the normal
setUp() if the one time setup initialization was done and if not just call
it.

So yeah, so far an unsatisfactory situation. I'm hoping Gabriel,
by the power of the unit testing book he ate, will find a better
solution.

If the above does not work I'm out of ideas, but I still don't understand why
it doesnt. Anyways, I'll give a try at the examples you sent tomorrow morning
and we can talk in real time.

cheers,

Gabriel

Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/java
one _______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4045,47fe8cab300815409313003!

Gabriel Roldán ha scritto:
...

Question: will we be able to do the setup in the base class alone? Or is
this something that individual tests will need to know about?

No, unfortunately it's not possible. With junit4 I did not manage to
get it fully working (one time tear down I cannot define, so the
subclass would have to manually invoke it), with junit3 you have
to define that suite() method and the comment it out each time
you have to run a single test (otherwise due to the suite() method
being there, you end up running them all anyways).

here is where I'm getting lost. I can swear I can run a single test case from ArcSdeDataStoreTest inside eclipse, even if it uses the junit3 "static suite() returns wrapper" approach, since suite() is not called at all when running a single test from inside eclipse, I just need to check in the normal setUp() if the one time setup initialization was done and if not just call it.

Don't know. Try for yourself with the project I sent. On my PC Eclipse
first run the chosen test, then runs all of the others as "unrooted
tests" (which it places in a subnode, never knew the junit view was
in fact a tree).

So yeah, so far an unsatisfactory situation. I'm hoping Gabriel,
by the power of the unit testing book he ate, will find a better
solution.

If the above does not work I'm out of ideas, but I still don't understand why it doesnt. Anyways, I'll give a try at the examples you sent tomorrow morning and we can talk in real time.

Excellent
Cheers
Andrea

okay,
to summarize:
1- we want non static one time setup and tear down
2- we want to be able to run a single test case from eclipse ui, but eclipse
does not cares and run all of the tests in the suite
3- we try to avoid switching to testNG, which would fill our expectations if
we choose the correct (outdated) version, but forces to get completely rid of
junit
4- we don't mind using Junit3 or Junit4 as long as it works

About 2), I found a reason, there's a bug in eclipse's Junit4 plugin when
running in Juni3 compatibility mode. If so, and you try to run a single test
case from a Junit3 test class, eclipse will run all the others and show them
under a tree node called "unrooted tests".

The solution (workaround) is to stick to Junit3 AND eclipse's Junit3 plugin.
This way it all works perfectly.

If nobody has a strong argument against that, that'd be it. Otherwise we're
all ears.

Cheers,

Gabriel

On Friday 11 April 2008 08:34:37 am Andrea Aime wrote:

Gabriel Roldán ha scritto:
...

>>> Question: will we be able to do the setup in the base class alone? Or
>>> is this something that individual tests will need to know about?
>>
>> No, unfortunately it's not possible. With junit4 I did not manage to
>> get it fully working (one time tear down I cannot define, so the
>> subclass would have to manually invoke it), with junit3 you have
>> to define that suite() method and the comment it out each time
>> you have to run a single test (otherwise due to the suite() method
>> being there, you end up running them all anyways).
>
> here is where I'm getting lost. I can swear I can run a single test case
> from ArcSdeDataStoreTest inside eclipse, even if it uses the junit3
> "static suite() returns wrapper" approach, since suite() is not called at
> all when running a single test from inside eclipse, I just need to check
> in the normal setUp() if the one time setup initialization was done and
> if not just call it.

Don't know. Try for yourself with the project I sent. On my PC Eclipse
first run the chosen test, then runs all of the others as "unrooted
tests" (which it places in a subnode, never knew the junit view was
in fact a tree).

>> So yeah, so far an unsatisfactory situation. I'm hoping Gabriel,
>> by the power of the unit testing book he ate, will find a better
>> solution.
>
> If the above does not work I'm out of ideas, but I still don't understand
> why it doesnt. Anyways, I'll give a try at the examples you sent tomorrow
> morning and we can talk in real time.

Excellent
Cheers
Andrea

!DSPAM:4045,47ff0687149841030819293!

Gabriel Roldán ha scritto:

okay,
to summarize:
1- we want non static one time setup and tear down
2- we want to be able to run a single test case from eclipse ui, but eclipse does not cares and run all of the tests in the suite
3- we try to avoid switching to testNG, which would fill our expectations if we choose the correct (outdated) version, but forces to get completely rid of junit
4- we don't mind using Junit3 or Junit4 as long as it works

About 2), I found a reason, there's a bug in eclipse's Junit4 plugin when running in Juni3 compatibility mode. If so, and you try to run a single test case from a Junit3 test class, eclipse will run all the others and show them under a tree node called "unrooted tests".

The solution (workaround) is to stick to Junit3 AND eclipse's Junit3 plugin. This way it all works perfectly.

If nobody has a strong argument against that, that'd be it. Otherwise we're all ears.

I've attached the latest version of that junit3 based thing.
Plan:
* have GeoServerAbstractTestSupport extend OneTimeSetupTest and
   implement the one time setup/teardown (any idea
   for a better name for the one time test supeclass? And no, I won't
   accept WWWTest as a candidate... no, TestWWW does not count either
   :) )
* change all tests setup/teardown to setupInternal/teardowninternal
* add the suite() method to the read only test classes

Hum, wondering... shall I roll up a GSIP or this set of mails
is enough?

Cheers
Andrea

Andrea Aime ha scritto:

I've attached the latest version of that junit3 based thing.

... this time with attachment
Cheers
Andrea

(attachments)

OneTimeSetupTest.java (3.38 KB)
OneTimeSetupTestTest.java (854 Bytes)

Andrea Aime ha scritto:

I've attached the latest version of that junit3 based thing.
Plan:
* have GeoServerAbstractTestSupport extend OneTimeSetupTest and
  implement the one time setup/teardown (any idea
  for a better name for the one time test supeclass? And no, I won't
  accept WWWTest as a candidate... no, TestWWW does not count either
  :) )
* change all tests setup/teardown to setupInternal/teardowninternal
* add the suite() method to the read only test classes

I've done a quick test by changing the geoserver classes.
The good: a sample test like GetCoverageTest goes down from 16 to 9
seconds.
The bad: the suite() method has to be defined in each class that
can run in "one time setup" mode
The ugly: every setup method has to be modified into setUpInternal,
and you have to remember not to call super.setUp() or an infinite
recursion triggers. We should come up with a way to throw an
exception if this happens and tell the user that he should
call super.setupInternal instead

Still wondering if I should turn this into a proposal.
Cheers
Andrea

On Friday 11 April 2008 06:50:47 pm Andrea Aime wrote:

Andrea Aime ha scritto:
> I've attached the latest version of that junit3 based thing.
> Plan:
> * have GeoServerAbstractTestSupport extend OneTimeSetupTest and
> implement the one time setup/teardown (any idea
> for a better name for the one time test supeclass? And no, I won't
> accept WWWTest as a candidate... no, TestWWW does not count either
>
> :slight_smile: )
>
> * change all tests setup/teardown to setupInternal/teardowninternal
> * add the suite() method to the read only test classes

I've done a quick test by changing the geoserver classes.
The good: a sample test like GetCoverageTest goes down from 16 to 9
seconds.
The bad: the suite() method has to be defined in each class that
can run in "one time setup" mode
The ugly: every setup method has to be modified into setUpInternal,
and you have to remember not to call super.setUp() or an infinite
recursion triggers. We should come up with a way to throw an
exception if this happens and tell the user that he should
call super.setupInternal instead

sounds like a good enough compromise to me.

Still wondering if I should turn this into a proposal.

I don't think so. We're not changing a core api nor adding new functionality.

my 2c.

Gabriel

Cheers
Andrea

!DSPAM:4045,47ff96ee122353327367457!