[Geoserver-devel] status of release scripts

GeoServer GeoServer Developer
1

Hi all,

I have been working on the scripts to automate our release process. Here is where things are at.

First, if you want to take a look at the scripts they are in my github repo in a branch called “build_scripts”:

https://github.com/jdeolive/geoserver/tree/build_scripts/build

Here is how they work. And I actually haven’t setup all the hudson jobs yet, but the underlying scripts are working. So here is how it will work.

On our main hudson server there is a job called “geoserver-release”:

http://hudson.opengeo.org/hudson/job/geoserver-release/

The job takes the following parameters:

  • branch to release from
  • revision to tag
  • release name

The underlying scripts actually take more parameters but i will get to those later. When run the above job does the following:

  1. creates the tag from the specified parameters
  2. checks out the tag and preps for release, updating version numbers, etc…
  3. performs the main release build
  4. assembles artifacts and uploads them to a distribution directory, currently located here:

http://gridlock.opengeo.org/geoserver/release/

  1. Fires off two installer jobs (one for windows, one for osx) on remote hudsons. The two hudsons are currently on our internal network so i can’t provide access to them.
  2. When the installer jobs complete they upload the artifacts to the same distribution directory specified above.

The end result is all the artifacts wind up in a single directory matching the release name. For example:

http://gridlock.opengeo.org/geoserver/release/2.2-beta2/

The above list of tasks is an overview, the main release script actually does a bit more which is what i would like to discuss. The script also checks jira and determines two things:

  1. is the release actually released on jira
  2. does the release have any open/unresolved issues filed against it

To do so it uses the jira rest api (which is nice btw). This is something I would like feedback on. We have a couple of options here on what to do with respect to jira.

  1. don’t do any checks against jira, and leave that a manual process
  2. halt the build if the release is not released in jira or there are unresolved issues, making releasing on a jira manually a precursor to running the script
  3. mark the release as released in jira, and push back any unresolved issues to the next version

More options welcome.

The second thing I would like to discuss is how we handle the README. Currently when we release we update the README on the main branch (and not the tag). I would kind of like to avoid committing anything to the main branch in an automated process. For example what happens if we run the release, it fails and we run it again. We have to check if the readme has been already updated, etc… While certainly possible with some script mojo it seems like an error prone process I would like to avoid.

So what I would like to propose is the following. We don’t ever update the README on the main release branch. We leave it generic with information about the project, where to find the docs, etc… On the tag we create we would generate a new file called “RELEASE_NOTES.txt” (or something to that effect) that contains the stuff we usually put in the README like revision info and the new and noteworthy info for that release. To generate the new and noteworthy for the release notes i was thinking we could come up with a heuristic. One such heuristic could be to pull out any new feature or improvement, and perhaps any issue that is a high priority one. Anyways, you get the idea. And naturally it would contain the link to the entire changelog from jira like we do now. Perhaps we should just stick to simple and put the revision info and the changelog link in the release notes. Probably what I will do for a first pass if people are into the idea.

Those are the two main things I need input on thus far. Aside from the above stuff the main thing left to do is to do the upload to sourceforge. I think this will be pretty straight forward. The idea is that there would be a second release job (called geoserver-release-publish or something) that would take the release tag as input and basically just upload all the artifacts from the distribution directory to sourceforge. The idea is that after the main release job (and the hudson installer jobs) complete the person doing the release would do a quick sanity check on the artifacts, fix issues, etc… Once completed the release publish job would do the final push.

Oh, and another TODO will be to take the associated geotools info (version, revision, etc…) and ensure that is what is included in the geoserver build. I guess same goes for geowebcache as well.

That is it for now. Looking forward to hearing feedback.

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

2

To do so it uses the jira rest api (which is nice btw). This is something I would like feedback on.

Bleck - tough call.

We have a couple of options here on what to do with respect to jira.

  1. don’t do any checks against jira, and leave that a manual process

In my experience going through the Jira list is educational; for two reasons:

  • as often I find items that are marked “resolved” but not yet closed as they are waiting confirmation.
  • often issues are fixed (usually due to issues being duplicated etc…)

This is also our “last chance” to see what is going on before the issues are punted over to the next release.

  1. halt the build if the release is not released in jira or there are unresolved issues, making releasing on a jira manually a precursor to running the script
  1. mark the release as released in jira, and push back any unresolved issues to the next version

I would go with this option (despite the temptation to halt the build for manual review).

Reason is:
a) we are shooting for a two hour release window
b) we can update the release notes in Jira after the fact; and the page will update

More options welcome.

Can we promote any "resolved’ issues to “closed” if they have now been included in a release?

Jody

3

On Wed, May 16, 2012 at 7:55 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

The end result is all the artifacts wind up in a single directory matching
the release name. For example:

http://gridlock.opengeo.org/geoserver/release/2.2-beta2/

Looking good :slight_smile:

The above list of tasks is an overview, the main release script actually
does a bit more which is what i would like to discuss. The script also
checks jira and determines two things:

1. is the release actually released on jira
2. does the release have any open/unresolved issues filed against it

To do so it uses the jira rest api (which is nice btw). This is something I
would like feedback on. We have a couple of options here on what to do with
respect to jira.

1. don't do any checks against jira, and leave that a manual process
2. halt the build if the release is not released in jira or there are
unresolved issues, making releasing on a jira manually a precursor to
running the script
3. mark the release as released in jira, and push back any unresolved
issues to the next version

Hum... I don't remember ever doing a release where all the tickets assigned to
it were resolved.
I'd go for 3, the release is not happening by itself anyways, and hopefully
there will be a bit of discussion on the ml before someone pushes the
"big red release button".

The second thing I would like to discuss is how we handle the README.
Currently when we release we update the README on the main branch (and not
the tag). I would kind of like to avoid committing anything to the main
branch in an automated process. For example what happens if we run the
release, it fails and we run it again. We have to check if the readme has
been already updated, etc... While certainly possible with some script mojo
it seems like an error prone process I would like to avoid.

So what I would like to propose is the following. We don't ever update the
README on the main release branch. We leave it generic with information
about the project, where to find the docs, etc... On the tag we create we
would generate a new file called "RELEASE_NOTES.txt" (or something to that
effect) that contains the stuff we usually put in the README like revision
info and the new and noteworthy info for that release. To generate the new
and noteworthy for the release notes i was thinking we could come up with a
heuristic. One such heuristic could be to pull out any new feature or
improvement, and perhaps any issue that is a high priority one. Anyways, you
get the idea. And naturally it would contain the link to the entire
changelog from jira like we do now. Perhaps we should just stick to simple
and put the revision info and the changelog link in the release notes.
Probably what I will do for a first pass if people are into the idea.

Hum... this file is going to be inside the packages and on sourceforge, but
wondering how many people are actually going to read it?
A link to the release notes on Jira seems reasonable to me.

Alternatively we allow the releaser to compose a narrative that is going to
be included in the README file, possibly the same message that is then
going to be sent to the mailing lists and reused in the blogs.

Those are the two main things I need input on thus far. Aside from the above
stuff the main thing left to do is to do the upload to sourceforge. I think
this will be pretty straight forward. The idea is that there would be a
second release job (called geoserver-release-publish or something) that
would take the release tag as input and basically just upload all the
artifacts from the distribution directory to sourceforge. The idea is that
after the main release job (and the hudson installer jobs) complete the
person doing the release would do a quick sanity check on the artifacts, fix
issues, etc... Once completed the release publish job would do the final
push.

Yep, sounds good to me.

Oh, and another TODO will be to take the associated geotools info (version,
revision, etc...) and ensure that is what is included in the geoserver
build. I guess same goes for geowebcache as well.

Yep

That is it for now. Looking forward to hearing feedback.

Nothing in particular. Kudos to your bash-fu and pretty excited about this
new work becoming real :slight_smile:

Cheers
Andrea

--
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

4

The second thing I would like to discuss is how we handle the README. Currently when we release we update the README on the main branch (and not the tag). I would kind of like to avoid committing anything to the main branch in an automated process. For example what happens if we run the release, it fails and we run it again. We have to check if the readme has been already updated, etc… While certainly possible with some script mojo it seems like an error prone process I would like to avoid.

So what I would like to propose is the following. We don’t ever update the README on the main release branch. We leave it generic with information about the project, where to find the docs, etc… On the tag we create we would generate a new file called “RELEASE_NOTES.txt” (or something to that effect) that contains the stuff we usually put in the README like revision info and the new and noteworthy info for that release. To generate the new and noteworthy for the release notes i was thinking we could come up with a heuristic. One such heuristic could be to pull out any new feature or improvement, and perhaps any issue that is a high priority one. Anyways, you get the idea. And naturally it would contain the link to the entire changelog from jira like we do now. Perhaps we should just stick to simple and put the revision info and the changelog link in the release notes. Probably what I will do for a first pass if people are into the idea.

I would look a lot better if you could generate a README.html using:

  • the boiler plate README from the main branch
  • idea: pull the release description from JIRA (if it is available ) and use it for the release specific description.
  • a link to the release notes in JIRA; optionally include a list of the “closed” items (as a highlights) and then offer a link through to JIRA for more information
  • (Wish) the features are often best summed up in the associated blog posts that have been made; could we troll the various geoserver, opengeo, geo-solutions blogs for a tag matching the release number - and provide links out to that information.

Thanks for the hard work on this justin; when I was on IRC I saw geotools patches still being applied - is the coast clear now?
Jody

5

You were very fast Justin :slight_smile:

Don’t have time right now to take a deep look at the scripts but I will do during the we as a personal interest.
Thanks for achieving this.

About JIRA, I also think option 3 is the best one.

Do you still need help to do something in order to close up this task?

Regards,
Alessio.

Ing. Alessio Fabiani
Founder / CTO GeoSolutions S.A.S.

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: (+39) 0584 96.23.13
fax: (+39) 0584 96.23.13
mobile:(+39) 331 62.33.686

http://www.geo-solutions.it
http://geo-solutions.blogspot.com
http://www.linkedin.com/in/alessiofabiani
https://twitter.com/alfa7961
http://twitter.com/geosolutions_it

On Wed, May 16, 2012 at 9:54 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

The second thing I would like to discuss is how we handle the README. Currently when we release we update the README on the main branch (and not the tag). I would kind of like to avoid committing anything to the main branch in an automated process. For example what happens if we run the release, it fails and we run it again. We have to check if the readme has been already updated, etc… While certainly possible with some script mojo it seems like an error prone process I would like to avoid.

So what I would like to propose is the following. We don’t ever update the README on the main release branch. We leave it generic with information about the project, where to find the docs, etc… On the tag we create we would generate a new file called “RELEASE_NOTES.txt” (or something to that effect) that contains the stuff we usually put in the README like revision info and the new and noteworthy info for that release. To generate the new and noteworthy for the release notes i was thinking we could come up with a heuristic. One such heuristic could be to pull out any new feature or improvement, and perhaps any issue that is a high priority one. Anyways, you get the idea. And naturally it would contain the link to the entire changelog from jira like we do now. Perhaps we should just stick to simple and put the revision info and the changelog link in the release notes. Probably what I will do for a first pass if people are into the idea.

I would look a lot better if you could generate a README.html using:

  • the boiler plate README from the main branch
  • idea: pull the release description from JIRA (if it is available ) and use it for the release specific description.
  • a link to the release notes in JIRA; optionally include a list of the “closed” items (as a highlights) and then offer a link through to JIRA for more information
  • (Wish) the features are often best summed up in the associated blog posts that have been made; could we troll the various geoserver, opengeo, geo-solutions blogs for a tag matching the release number - and provide links out to that information.

Thanks for the hard work on this justin; when I was on IRC I saw geotools patches still being applied - is the coast clear now?

Jody

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

6

On Wed, May 16, 2012 at 1:48 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

To do so it uses the jira rest api (which is nice btw). This is something I would like feedback on.

Bleck - tough call.

We have a couple of options here on what to do with respect to jira.

  1. don’t do any checks against jira, and leave that a manual process

In my experience going through the Jira list is educational; for two reasons:

  • as often I find items that are marked “resolved” but not yet closed as they are waiting confirmation.
  • often issues are fixed (usually due to issues being duplicated etc…)

This is also our “last chance” to see what is going on before the issues are punted over to the next release.

  1. halt the build if the release is not released in jira or there are unresolved issues, making releasing on a jira manually a precursor to running the script
  1. mark the release as released in jira, and push back any unresolved issues to the next version

I would go with this option (despite the temptation to halt the build for manual review).

Reason is:
a) we are shooting for a two hour release window
b) we can update the release notes in Jira after the fact; and the page will update

More options welcome.

Can we promote any "resolved’ issues to “closed” if they have now been included in a release?

I don’t see why not… I believe resolved is meant to signify it has been fixed by a developer and closed is used more as a confirmation that the fix actually works. We don’t really use the latter so i am ok with just marking stuff with closed when we release. Like to hear what others thing about this before proceeding though.

Jody

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

7

On Wed, May 16, 2012 at 1:53 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, May 16, 2012 at 7:55 AM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

The end result is all the artifacts wind up in a single directory matching
the release name. For example:

http://gridlock.opengeo.org/geoserver/release/2.2-beta2/

Looking good :slight_smile:

The above list of tasks is an overview, the main release script actually
does a bit more which is what i would like to discuss. The script also
checks jira and determines two things:

  1. is the release actually released on jira
  2. does the release have any open/unresolved issues filed against it

To do so it uses the jira rest api (which is nice btw). This is something I
would like feedback on. We have a couple of options here on what to do with
respect to jira.

  1. don’t do any checks against jira, and leave that a manual process
  2. halt the build if the release is not released in jira or there are
    unresolved issues, making releasing on a jira manually a precursor to
    running the script
  3. mark the release as released in jira, and push back any unresolved
    issues to the next version

Hum… I don’t remember ever doing a release where all the tickets assigned to
it were resolved.
I’d go for 3, the release is not happening by itself anyways, and hopefully
there will be a bit of discussion on the ml before someone pushes the
“big red release button”.

To clarify i wasn’t implying that we fix all issues assigned to the version being released, just move them. The moving being manual or automatic. Sounds like people like the automatic approach so will push on with that.

The second thing I would like to discuss is how we handle the README.
Currently when we release we update the README on the main branch (and not
the tag). I would kind of like to avoid committing anything to the main
branch in an automated process. For example what happens if we run the
release, it fails and we run it again. We have to check if the readme has
been already updated, etc… While certainly possible with some script mojo
it seems like an error prone process I would like to avoid.

So what I would like to propose is the following. We don’t ever update the
README on the main release branch. We leave it generic with information
about the project, where to find the docs, etc… On the tag we create we
would generate a new file called “RELEASE_NOTES.txt” (or something to that
effect) that contains the stuff we usually put in the README like revision
info and the new and noteworthy info for that release. To generate the new
and noteworthy for the release notes i was thinking we could come up with a
heuristic. One such heuristic could be to pull out any new feature or
improvement, and perhaps any issue that is a high priority one. Anyways, you
get the idea. And naturally it would contain the link to the entire
changelog from jira like we do now. Perhaps we should just stick to simple
and put the revision info and the changelog link in the release notes.
Probably what I will do for a first pass if people are into the idea.

Hum… this file is going to be inside the packages and on sourceforge, but
wondering how many people are actually going to read it?
A link to the release notes on Jira seems reasonable to me.

Cool. I think the same… it would be interesting to know how many people actually do read that file.

Alternatively we allow the releaser to compose a narrative that is going to
be included in the README file, possibly the same message that is then
going to be sent to the mailing lists and reused in the blogs.

Yeah, i thought about this as well. I will look into some of the options here. Good point about using it for the notifications as well.

Those are the two main things I need input on thus far. Aside from the above
stuff the main thing left to do is to do the upload to sourceforge. I think
this will be pretty straight forward. The idea is that there would be a
second release job (called geoserver-release-publish or something) that
would take the release tag as input and basically just upload all the
artifacts from the distribution directory to sourceforge. The idea is that
after the main release job (and the hudson installer jobs) complete the
person doing the release would do a quick sanity check on the artifacts, fix
issues, etc… Once completed the release publish job would do the final
push.

Yep, sounds good to me.

Oh, and another TODO will be to take the associated geotools info (version,
revision, etc…) and ensure that is what is included in the geoserver
build. I guess same goes for geowebcache as well.

Yep

That is it for now. Looking forward to hearing feedback.

Nothing in particular. Kudos to your bash-fu and pretty excited about this
new work becoming real :slight_smile:

Cheers
Andrea

–
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

8

On Wed, May 16, 2012 at 1:54 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

The second thing I would like to discuss is how we handle the README. Currently when we release we update the README on the main branch (and not the tag). I would kind of like to avoid committing anything to the main branch in an automated process. For example what happens if we run the release, it fails and we run it again. We have to check if the readme has been already updated, etc… While certainly possible with some script mojo it seems like an error prone process I would like to avoid.

So what I would like to propose is the following. We don’t ever update the README on the main release branch. We leave it generic with information about the project, where to find the docs, etc… On the tag we create we would generate a new file called “RELEASE_NOTES.txt” (or something to that effect) that contains the stuff we usually put in the README like revision info and the new and noteworthy info for that release. To generate the new and noteworthy for the release notes i was thinking we could come up with a heuristic. One such heuristic could be to pull out any new feature or improvement, and perhaps any issue that is a high priority one. Anyways, you get the idea. And naturally it would contain the link to the entire changelog from jira like we do now. Perhaps we should just stick to simple and put the revision info and the changelog link in the release notes. Probably what I will do for a first pass if people are into the idea.

I would look a lot better if you could generate a README.html using:

  • the boiler plate README from the main branch
  • idea: pull the release description from JIRA (if it is available ) and use it for the release specific description.
  • a link to the release notes in JIRA; optionally include a list of the “closed” items (as a highlights) and then offer a link through to JIRA for more information
  • (Wish) the features are often best summed up in the associated blog posts that have been made; could we troll the various geoserver, opengeo, geo-solutions blogs for a tag matching the release number - and provide links out to that information.

Thanks for the hard work on this justin; when I was on IRC I saw geotools patches still being applied - is the coast clear now?

Yup, had a few patches yesterday i wanted to get in, all good to go know though.

Jody

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.