[Geoserver-users] Discussing a switch to a time boxed release model

GeoServer GeoServer User
1

Hi all,
there is a set of recent event that makes me wonder if we would be better off
switching from the current “ad hoc” release system to a time released model
with more frequent releases.

The events that I’m thinking about are:

  • Jody back in December was ready to release GeoTools. But other people
    were not of the same advise and were actually surprised about release
    talks in the first place
  • me and Ben were ready to start the betas in late January, but others
    were not
  • in May with a beta out and we are still talking through GSIPs

Many “big” projects (linux distros, Gnome and so on) are using a timed
released model, so I’m wondering if we can try a similar approach with
GeoServer.

Just for the sake of discussion I’ve tried to lay out a GeoServer relase plan
based on a time boxed approach, here it is:

Inline image 4

Main points in the above proposed cycle:

  • monthly releases
  • six months release cycle
  • the stable series (yellow-is area) is mostly geared towards bug fixing and new plugins,
    with some agreement in the PSC we can also also backport new stuff
    that requires altering the published API
  • the green area is free development for whatever you want, with of course the
    requirement that the software keeps on building fine
  • the red area is hardening, which starts when beta1 is released and disallows
    new features and new API focusing on bug fixing
  • a new trunk is cut as soon as we enter harderning, that is, the day we release beta2
  • we get six months in which can still do pretty much whatever we want (well, sorta,
    I would expect no revolutions once beta1 is out)

The downsides that need to be tamed:

  • we have three months with 3 active branches
  • the process is release happy

About the 3 active branches, what we could do is maybe to cut the new trunk
one month later (at beta2), and my observation is that backporting stuff with
git “cherry-pick” is normally less painful, so if we also switch fully to git that
should be less of a pain.

About being relase happy, we need to make the gt/gs synched up releases
a snap, something that can be done in 2-4 hours tops, and something that
more people can do.

I believe the current approach, in which we take a well known revision that
already passed the full builds and the CITE tests on the build server, already
cuts down lots of the pain.
One more thing we could do could be to have a manually started build
on the server that automates further the release process, some parametric
Hudson/Jenkins build that taken a revision number creates the tag, switches
the revision numbers, commits the changes and builds the new binaries
making them available in some location that allows a download of them
(one for geotools, one for geoserver), so that the release technician
can download and double check, and then another one that automates
the uploading to sourceforge (assuming the hudson server has
plenty of bandwidth).

This would allow the release technician to focus on the other mundane
tasks:

  • manual testing the geotools builds (actually this one too could be
    automated, just build against an empty repo)
  • manual testing geoserver, building installers for various platforms
  • releasing on jira, preparing the announcements, blogs

The above too should be something that more people can do (e.g.,
some colleague of mine at GeoSolutions that does not normally
participate in active development), which means we would have to give
admin access to jira to more people

This way, when “release day” comes nobody is surprised, the
release can be cut in a rather quick time, and we can share the releases
load with more people.

Well… this is it. Opinions?

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

2

In general I think it looks great, a few things though. I think given the current effort to put out releases 1 month is probably asking a bit much given the resources we have on the project. So I think to do one month cycles we really do need to better automate our release process with a hudson job that does most of the work.

It would also be good to have some better defined (and perhaps stricter) guidelines about what is acceptable to commit given the current phase of an iteration. For instance obviously once we move to a stable or hardening phase GSIP’s that drastically alter the core are out of the question. It would be good if we had a more concrete definition of “drastically alter the core”. Like should we be strict and say stable/hardening means strictly only bug fixes? With a faster release cycle it could make more sense to have stricter guidelines since if you don’t get something into this release there is one not too far off. This is exactly why we ran into the fluerry of gsip issue… with another release 1.5 years away it certainly puts the pressure on to cram stuff in.

Anyways, great stuff. I like where this is going, big things from my standpoint.

  1. better automation of release, which i am happy to help with
  2. better guidelines for what type of development is acceptable during what phases

-Justin

On Tue, May 1, 2012 at 2:34 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi all,
there is a set of recent event that makes me wonder if we would be better off
switching from the current “ad hoc” release system to a time released model
with more frequent releases.

The events that I’m thinking about are:

  • Jody back in December was ready to release GeoTools. But other people
    were not of the same advise and were actually surprised about release
    talks in the first place
  • me and Ben were ready to start the betas in late January, but others
    were not
  • in May with a beta out and we are still talking through GSIPs

Many “big” projects (linux distros, Gnome and so on) are using a timed
released model, so I’m wondering if we can try a similar approach with
GeoServer.

Just for the sake of discussion I’ve tried to lay out a GeoServer relase plan
based on a time boxed approach, here it is:

Inline image 4

Main points in the above proposed cycle:

  • monthly releases
  • six months release cycle
  • the stable series (yellow-is area) is mostly geared towards bug fixing and new plugins,
    with some agreement in the PSC we can also also backport new stuff
    that requires altering the published API
  • the green area is free development for whatever you want, with of course the
    requirement that the software keeps on building fine
  • the red area is hardening, which starts when beta1 is released and disallows
    new features and new API focusing on bug fixing
  • a new trunk is cut as soon as we enter harderning, that is, the day we release beta2
  • we get six months in which can still do pretty much whatever we want (well, sorta,
    I would expect no revolutions once beta1 is out)

The downsides that need to be tamed:

  • we have three months with 3 active branches
  • the process is release happy

About the 3 active branches, what we could do is maybe to cut the new trunk
one month later (at beta2), and my observation is that backporting stuff with
git “cherry-pick” is normally less painful, so if we also switch fully to git that
should be less of a pain.

About being relase happy, we need to make the gt/gs synched up releases
a snap, something that can be done in 2-4 hours tops, and something that
more people can do.

I believe the current approach, in which we take a well known revision that
already passed the full builds and the CITE tests on the build server, already
cuts down lots of the pain.
One more thing we could do could be to have a manually started build
on the server that automates further the release process, some parametric
Hudson/Jenkins build that taken a revision number creates the tag, switches
the revision numbers, commits the changes and builds the new binaries
making them available in some location that allows a download of them
(one for geotools, one for geoserver), so that the release technician
can download and double check, and then another one that automates
the uploading to sourceforge (assuming the hudson server has
plenty of bandwidth).

This would allow the release technician to focus on the other mundane
tasks:

  • manual testing the geotools builds (actually this one too could be
    automated, just build against an empty repo)
  • manual testing geoserver, building installers for various platforms
  • releasing on jira, preparing the announcements, blogs

The above too should be something that more people can do (e.g.,
some colleague of mine at GeoSolutions that does not normally
participate in active development), which means we would have to give
admin access to jira to more people

This way, when “release day” comes nobody is surprised, the
release can be cut in a rather quick time, and we can share the releases
load with more people.

Well… this is it. Opinions?

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoTools-Devel mailing list
GeoTools-Devel@anonymised.coms.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

3

On Tue, May 1, 2012 at 5:54 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

In general I think it looks great, a few things though. I think given the current effort to put out releases 1 month is probably asking a bit much given the resources we have on the project. So I think to do one month cycles we really do need to better automate our release process with a hudson job that does most of the work.

It would also be good to have some better defined (and perhaps stricter) guidelines about what is acceptable to commit given the current phase of an iteration. For instance obviously once we move to a stable or hardening phase GSIP’s that drastically alter the core are out of the question. It would be good if we had a more concrete definition of “drastically alter the core”. Like should we be strict and say stable/hardening means strictly only bug fixes? With a faster release cycle it could make more sense to have stricter guidelines since if you don’t get something into this release there is one not too far off. This is exactly why we ran into the fluerry of gsip issue… with another release 1.5 years away it certainly puts the pressure on to cram stuff in.

Anyways, great stuff. I like where this is going, big things from my standpoint.

  1. better automation of release, which i am happy to help with
  2. better guidelines for what type of development is acceptable during what phases

Fully agree on the higher automation (tried to discuss some ideas about it in my
original mail).

About what is acceptable and not, what about the following:

  • stable series: only bug fixes and new features that do not require API changes
    or large patches to existing systems (that is, if you are contributing a new module
    the patch can be as large as you want, but a “bug fix” that rewrites half of WFS
    is not welcomed unless the PSC really really wants such change badly in)
  • trunk: free reign, but large changes still need a GSIP and reviews
  • hardening: no new features, only bug fixing to make sure people concentrate on
    that, if you have something new it has to go on trunk for the time being

The above should be, imho, taken more or less as strict rules that the PSC should
try to enforce (the above, or whatever we come up with).
That said, the PSC should be allowed to decide outside of the above rules
in case of dire necessity (e.g., something that could threaten or damage the
project severely if not done outside of the rules).

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

4

On Wed, May 2, 2012 at 9:27 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Tue, May 1, 2012 at 5:54 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

In general I think it looks great, a few things though. I think given the current effort to put out releases 1 month is probably asking a bit much given the resources we have on the project. So I think to do one month cycles we really do need to better automate our release process with a hudson job that does most of the work.

It would also be good to have some better defined (and perhaps stricter) guidelines about what is acceptable to commit given the current phase of an iteration. For instance obviously once we move to a stable or hardening phase GSIP’s that drastically alter the core are out of the question. It would be good if we had a more concrete definition of “drastically alter the core”. Like should we be strict and say stable/hardening means strictly only bug fixes? With a faster release cycle it could make more sense to have stricter guidelines since if you don’t get something into this release there is one not too far off. This is exactly why we ran into the fluerry of gsip issue… with another release 1.5 years away it certainly puts the pressure on to cram stuff in.

Anyways, great stuff. I like where this is going, big things from my standpoint.

  1. better automation of release, which i am happy to help with
  2. better guidelines for what type of development is acceptable during what phases

Fully agree on the higher automation (tried to discuss some ideas about it in my
original mail).

About what is acceptable and not, what about the following:

  • stable series: only bug fixes and new features that do not require API changes
    or large patches to existing systems (that is, if you are contributing a new module
    the patch can be as large as you want, but a “bug fix” that rewrites half of WFS
    is not welcomed unless the PSC really really wants such change badly in)
  • trunk: free reign, but large changes still need a GSIP and reviews
  • hardening: no new features, only bug fixing to make sure people concentrate on
    that, if you have something new it has to go on trunk for the time being

These sounds reasonable to me. Obviously nothing is going to be black and white but they are a good start and we can refine as need be. As part of this proposal we should ensure they are described in the developer guide. Having this rules written down somewhere we can point to easily will be more effective than just coming up with a common unoffocial understanding among PSC and core developers.

The above should be, imho, taken more or less as strict rules that the PSC should
try to enforce (the above, or whatever we come up with).
That said, the PSC should be allowed to decide outside of the above rules
in case of dire necessity (e.g., something that could threaten or damage the
project severely if not done outside of the rules).

Certainly, the PSC should be able to make exceptions on a case by case basis. Again it might be nice to enumerate what those are. In the past as far as I know the biggest one has been that a client needs a significant new feature, but needs it on a stable branch.

Cheers

Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

I don’t mind switching to a different release model.

My only thought is that what we need (rather then a model) is a release; and the commitment from project sponsors to make it happen. I suspect the biggest point in favour of a time released model is the predictability that would lend for planning resourcing.

Six months is fine; I was also happy with the yearly timeframe we were hitting if that is easier to manage. I encourage everyone to talk to their employer before voting on this one.

Well… this is it. Opinions?

Thanks for the detailed, and thoughtful email - my only comment is lets go!

(And yes I will talk to my employer; although I won’t hold my breath until after government year end. As a volunteer I will be find releasing GeoTools if we can find others to tame GeoServer and its Cite tests).

Jody