[GeoNetwork-devel] Continuous Integration builds [SEC=UNCLASSIFIED]

GeoNetwork GeoNetwork Developer
1

Hi James,

Are you aware of the major differences between the trunk/build.xml and BlueNetMEST/build.xml?

As I am primary author of BlueNet/build.xml and will shortly be departing Geoscience Australia I should probably provide the group with some insight into why the differences exist and what I was hoping to achieve by making them.

The trunk/build.xml has the following advantages:

  1. short

  2. simple

  3. works!

Is disadvantages are:

  1. compiles Java to .build (directory hidden on some OS’s)

  2. deletes .build at end of targets (causes problems in tools like Eclipse, .class files removed)

  3. performs ‘in place’ updates of generated jar files (updates ‘source’ tree, potential pollution of SVN repository if checked in)

It was the in place updates that prompted me to rework the BlueNet build. Rather than use .build as the location for classes a build directory was introduced. Changing the name unsures the directory is not hidden. The build directory is kept at build end (unless explicitly removed by calling the clean target). I have Eclipse and Ant happily sharing the build directory, no more tool issues.

I feel quite strongly about mixing source files with generated content. It generally makes a real mess of any repository and is an opportunity for library version issues to creep in. The dist directory was introduced to separate source files from generated content. Geonetwork.jar and other files are built here. Static content from web/geonetwork (and others) are copied here and then merged with the generated JAR files. The contents of dist/web/* are deployable and this is where I have Tomcat configured to run from. It also has the effect of keeping the server logs clear of the source tree, a problem you seem to have encountered.

Building this way does have disadvantages:

  1. It breaks ‘hot edit’, a server running against dist is working with a copy of the source files. Editing these files yields immediate changes in the website but files are overwritten in the next build (developer must copy back changes)

  2. The build itself becomes more complex, more targets are used

  3. The Gast utility breaks as it has a classloader structure hardcoded to the old build structure (solvable, use Gast from the installer)

Future Ideas

Move dependent JAR files from individual application WEB-INF/lib directories to a common lib directory. The intent of this is to remove jar file duplication among the three GN applications and simplify the upgrading of third party libraries.

The use of Maven has been mentioned. If people decide not to use Maven then I would suggest the introduction of Apache Ivy. This will force developers to document

  1. direct dependencies on third party libraries (and their context, be it runtime, build or test)

  2. clear indication of transitive dependencies (i.e. third party library requires another library)

It will also highlight any library incompatibilities, a common problem with complex projects that use numerous jars.

More work is required on the gast utility, the way it loads classes needs to be reviewed.

Regards,

Stephen

-----Original Message-----
From: James.Dempsey@anonymised.com [mailto:James.Dempsey@anonymised.com]
Sent: Wednesday, 15 October 2008 5:42
To: Jeroen.Ticheler@anonymised.com.; geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] Continuous Integration builds

Hi Jeroen,

Based on the failure message it sent out after my commit I think the BlueNetMEST build is running installer/build.xml rather than build.xml. Alternatively it could be running build.xml from the root folder with the all or installer targets, but that should be doing the war build also. Without seeing the full output it is difficult to tell which is the case.

Cheers,

James Dempsey

From: Jeroen Ticheler [mailto:Jeroen.Ticheler@anonymised.com]
Sent: Wednesday, 15 October 2008 8:24 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] Continuous Integration builds

Hi all,

I set up Continuous integration for three GeoNetwork builds now:

  • Trunk

  • ebXML related developments under the 3.0.x branch

  • BlueNetMEST branch

You can find the build status page at: http://buildtest.osgeo.org/osgeo/geonetwork/hudson/

The BlueNetMEST build failed the first time due to a missing file that’s actually generated at runtime. Not sure why, but leave it up to Simon to look into it.

Hope this is helpful. An email will be send to the commit mailinglist and to the committer in case a build breaks due to a new commit. The system does an hourly check for new commits.

Cheers,

Jeroen

Jeroen Ticheler

GeoCat bv
Grotenhuisweg 61

7384 CT Wilp

Tel: +31 (0)6 81286572

http://geocat.net

Please consider the environment before printing this email.

2

Hi Stephen,

Thanks for that run down - I must admit I’ve been focussing on the BlueNetMEST version only and haven’t looked at trunk too much.

While I had some fun initially migrating my Eclipse workspace to catch up with new build structure I have to say I like it a lot. It does the job of keeping things separate (something which I too am very keen on) and makes deployment to Tomcat quite simple. To that end I have added some war file generation tasks locally which I was intending on contributing if no one objects.

The other thing it does is provides an excellent stepping stone to get to Maven or Ivy. When I first looked at GN, I was at a loss for how you might get to something that Maven could cope with. With the clearer output structure that your build produces it seems a bit more achievable. I still don’t think it will be a trivial conversion though :slight_smile:

Cheers,
James Dempsey

From: Stephen.Davies@anonymised.com [mailto:Stephen.Davies@anonymised.com]
Sent: Thursday, 16 October 2008 11:00 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] Continuous Integration builds [SEC=UNCLASSIFIED]

Hi James,

Are you aware of the major differences between the trunk/build.xml and BlueNetMEST/build.xml?

As I am primary author of BlueNet/build.xml and will shortly be departing Geoscience Australia I should probably provide the group with some insight into why the differences exist and what I was hoping to achieve by making them.

The trunk/build.xml has the following advantages:

  1. short

  2. simple

  3. works!

Is disadvantages are:

  1. compiles Java to .build (directory hidden on some OS’s)

  2. deletes .build at end of targets (causes problems in tools like Eclipse, .class files removed)

  3. performs ‘in place’ updates of generated jar files (updates ‘source’ tree, potential pollution of SVN repository if checked in)

It was the in place updates that prompted me to rework the BlueNet build. Rather than use .build as the location for classes a build directory was introduced. Changing the name unsures the directory is not hidden. The build directory is kept at build end (unless explicitly removed by calling the clean target). I have Eclipse and Ant happily sharing the build directory, no more tool issues.

I feel quite strongly about mixing source files with generated content. It generally makes a real mess of any repository and is an opportunity for library version issues to creep in. The dist directory was introduced to separate source files from generated content. Geonetwork.jar and other files are built here. Static content from web/geonetwork (and others) are copied here and then merged with the generated JAR files. The contents of dist/web/* are deployable and this is where I have Tomcat configured to run from. It also has the effect of keeping the server logs clear of the source tree, a problem you seem to have encountered.

Building this way does have disadvantages:

  1. It breaks ‘hot edit’, a server running against dist is working with a copy of the source files. Editing these files yields immediate changes in the website but files are overwritten in the next build (developer must copy back changes)

  2. The build itself becomes more complex, more targets are used

  3. The Gast utility breaks as it has a classloader structure hardcoded to the old build structure (solvable, use Gast from the installer)

Future Ideas

Move dependent JAR files from individual application WEB-INF/lib directories to a common lib directory. The intent of this is to remove jar file duplication among the three GN applications and simplify the upgrading of third party libraries.

The use of Maven has been mentioned. If people decide not to use Maven then I would suggest the introduction of Apache Ivy. This will force developers to document

  1. direct dependencies on third party libraries (and their context, be it runtime, build or test)

  2. clear indication of transitive dependencies (i.e. third party library requires another library)

It will also highlight any library incompatibilities, a common problem with complex projects that use numerous jars.

More work is required on the gast utility, the way it loads classes needs to be reviewed.

Regards,

Stephen

-----Original Message-----
From: James.Dempsey@anonymised.com [mailto:James.Dempsey@anonymised.com]
Sent: Wednesday, 15 October 2008 5:42
To: Jeroen.Ticheler@anonymised.com; geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] Continuous Integration builds

Hi Jeroen,

Based on the failure message it sent out after my commit I think the BlueNetMEST build is running installer/build.xml rather than build.xml. Alternatively it could be running build.xml from the root folder with the all or installer targets, but that should be doing the war build also. Without seeing the full output it is difficult to tell which is the case.

Cheers,

James Dempsey

From: Jeroen Ticheler [mailto:Jeroen.Ticheler@anonymised.com]
Sent: Wednesday, 15 October 2008 8:24 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] Continuous Integration builds

Hi all,

I set up Continuous integration for three GeoNetwork builds now:

  • Trunk

  • ebXML related developments under the 3.0.x branch

  • BlueNetMEST branch

You can find the build status page at: http://buildtest.osgeo.org/osgeo/geonetwork/hudson/

The BlueNetMEST build failed the first time due to a missing file that’s actually generated at runtime. Not sure why, but leave it up to Simon to look into it.

Hope this is helpful. An email will be send to the commit mailinglist and to the committer in case a build breaks due to a new commit. The system does an hourly check for new commits.

Cheers,

Jeroen

Jeroen Ticheler

GeoCat bv
Grotenhuisweg 61

7384 CT Wilp

Tel: +31 (0)6 81286572

http://geocat.net

Please consider the environment before printing this email.