[SAC] Resurrect continuous integration VM (once again)

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well, especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org

On Tue, Sep 20, 2011 at 11:44:44AM +0100, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely
well, especially for the C camp, before it disappeared.
I want to bring it back.

Welcome !

I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

Ouch, for a moment I belived you changed your mind about moving to Java.
For those (like me) that didn't know:
"jenkins-ci" is the new name for "hudson".

What are the pros/cons for droppin Buildbot again ?
What does it take to donate build slaves ?

Btw, taking a look at the packages list available on my systems (from
debian lenny to ubuntu 10.04) to check availability status of
jenkings/hudson/buildbot packages, I've stumbled upon
http://bitten.edgewall.org/ which naturally integrates into trac.
Does anyone have experience with it ?
("buildbot" and "bitten" were the only available packages, for the record).

--strk;

  () Free GIS & Flash consultant/developer
  /\ strk's services

On 20/09/11 13:33, Sandro Santilli wrote:

On Tue, Sep 20, 2011 at 11:44:44AM +0100, Mateusz Loskot wrote:

I discussed this shortly with Howard and Chris in Denver and we
agreed to replace old-good Buildbot with http://jenkins-ci.org

Ouch, for a moment I belived you changed your mind about moving to
Java.

Let's say, I had been undecided :slight_smile:

For those (like me) that didn't know: "jenkins-ci" is the new name
for "hudson".

Yes.

What are the pros/cons for droppin Buildbot again ?

The major one is that it requires maintainer to be familiar with
low-level Buildbot-Fu. I seem to be the only person here who
understands the configuration [1] and is able to maintain it
[1] http://svn.osgeo.org/osgeo/buildbot/
This is a disadvantage.
The Jenkins is more user and maintainer friendly, is easier to
configure, connect new slaves, etc.

When I was testing hudson, I didn't like several things, that's why I
couldn't make final decision. Here we go on Hudson (versus my Buildbot
experience):
- documentation was really bad, based on Wiki mess
- (So,) I could not find any "best practice" setup to figure out how to
configure it best for OSGeo
- (So,) I had impression it's very to make setup mess (easy access &
trivial operations leading to countless slaves and jobs leading to
multi-dimensional configuration extreme)

Buildbot has fantastic documentation, indeed.

Long story short, both Buildbot and Jenkins are very powerful and
customisable machines. But, from maintainer point of view - trying to
find analogy in world of data processing - the former is like manually
maintaining large and complex GDAL VRT files, the latter is like click &
drag in GUI of Safe FME software.

What does it take to donate build slaves ?

Are you asking about this?

https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds

Btw, taking a look at the packages list available on my systems
(from debian lenny to ubuntu 10.04) to check availability status of
jenkings/hudson/buildbot packages, I've stumbled upon
http://bitten.edgewall.org/ which naturally integrates into trac.
Does anyone have experience with it ? ("buildbot" and "bitten" were
the only available packages, for the record).

Trac is a really great idea, but very badly implemented.
I'd like keep my own hands off the Trac.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org

On Sep 20, 2011, at 9:07 AM, ext Mateusz Loskot wrote:

On 20/09/11 13:33, Sandro Santilli wrote:

On Tue, Sep 20, 2011 at 11:44:44AM +0100, Mateusz Loskot wrote:

I discussed this shortly with Howard and Chris in Denver and we
agreed to replace old-good Buildbot with http://jenkins-ci.org

Ouch, for a moment I belived you changed your mind about moving to
Java.

Let's say, I had been undecided :slight_smile:

For those (like me) that didn't know: "jenkins-ci" is the new name
for "hudson".

Yes.

What are the pros/cons for droppin Buildbot again ?

A pro for Jenkins is "some people who use SAC know it, and use it." I
maintain our Jenkins server at work, which runs with a half dozen slaves
and runs 80+ jobs; I won't presume to be a Jenkins master, and it certainly
has its downsides, but it really does make for a very nice tool with relatively
simple administration, even for someone who is not a Java wizard (like me).

I am willing to help set up and support a Jenkins install for OSGeo,
as a member of SAC.

I can't say the same about Buildbot.

I think overall, Jenkins will be better for most of the use cases I can imagine
for OSGeo continuous building.

-- Chris

The major one is that it requires maintainer to be familiar with
low-level Buildbot-Fu. I seem to be the only person here who
understands the configuration [1] and is able to maintain it
[1] http://svn.osgeo.org/osgeo/buildbot/
This is a disadvantage.
The Jenkins is more user and maintainer friendly, is easier to
configure, connect new slaves, etc.

When I was testing hudson, I didn't like several things, that's why I
couldn't make final decision. Here we go on Hudson (versus my Buildbot
experience):
- documentation was really bad, based on Wiki mess
- (So,) I could not find any "best practice" setup to figure out how to
configure it best for OSGeo
- (So,) I had impression it's very to make setup mess (easy access &
trivial operations leading to countless slaves and jobs leading to
multi-dimensional configuration extreme)

Buildbot has fantastic documentation, indeed.

Long story short, both Buildbot and Jenkins are very powerful and
customisable machines. But, from maintainer point of view - trying to
find analogy in world of data processing - the former is like manually
maintaining large and complex GDAL VRT files, the latter is like click &
drag in GUI of Safe FME software.

What does it take to donate build slaves ?

Are you asking about this?

https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds

Btw, taking a look at the packages list available on my systems
(from debian lenny to ubuntu 10.04) to check availability status of
jenkings/hudson/buildbot packages, I've stumbled upon
http://bitten.edgewall.org/ which naturally integrates into trac.
Does anyone have experience with it ? ("buildbot" and "bitten" were
the only available packages, for the record).

Trac is a really great idea, but very badly implemented.
I'd like keep my own hands off the Trac.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

On 20/09/11 11:44, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well,
especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

------------------------------------^^^^^^^^^^^^^^^^

Bump.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org

On Thu, Oct 6, 2011 at 1:06 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

On 20/09/11 11:44, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well,
especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

------------------------------------^^^^^^^^^^^^^^^^

Bump.

Mateusz,

I'm concerned that our two servers at OSU OSL are getting
rather crowded with VMs and that it might be hard to
host a big build-and-test VM there right now. I wonder
if John might be able to provide an appropriate VM at
Telascience?

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

On 06/10/11 21:08, Frank Warmerdam wrote:

On Thu, Oct 6, 2011 at 1:06 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

On 20/09/11 11:44, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well,
especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

------------------------------------^^^^^^^^^^^^^^^^

Bump.

Mateusz,

I'm concerned that our two servers at OSU OSL are getting
rather crowded with VMs and that it might be hard to
host a big build-and-test VM there right now. I wonder
if John might be able to provide an appropriate VM at
Telascience?

I see. Let's wait and see what's the situation when the dust after
current motions/migrations settles down.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org

On Thu, Oct 06, 2011 at 09:26:30PM +0100, Mateusz Loskot wrote:

On 06/10/11 21:08, Frank Warmerdam wrote:

...

> I'm concerned that our two servers at OSU OSL are getting
> rather crowded with VMs and that it might be hard to
> host a big build-and-test VM there right now. I wonder
> if John might be able to provide an appropriate VM at
> Telascience?

I see. Let's wait and see what's the situation when the dust after
current motions/migrations settles down.

Mat, what about starting by gathering informations about required
dependencies and resources for the build master ?

I think it would be a good service even if the machine does only the
master side. We want slaves to be distributed as much as possible
(thus my inquiry about how widely distributed hudson was vs. buildbot..).

--strk;

  () Free GIS & Flash consultant/developer
  /\ strk's services

On 07/10/11 10:22, Sandro Santilli wrote:

On Thu, Oct 06, 2011 at 09:26:30PM +0100, Mateusz Loskot wrote:

On 06/10/11 21:08, Frank Warmerdam wrote:

...

I'm concerned that our two servers at OSU OSL are getting
rather crowded with VMs and that it might be hard to
host a big build-and-test VM there right now. I wonder
if John might be able to provide an appropriate VM at
Telascience?

I see. Let's wait and see what's the situation when the dust after
current motions/migrations settles down.

Mat, what about starting by gathering informations about required
dependencies and resources for the build master ?

I'm supportive.

I think it would be a good service even if the machine does only the
master side. We want slaves to be distributed as much as possible

Yes. The master machine is quite solid two-cores and 4GB RAM machine.
Then we add remote slave machines.

(thus my inquiry about how widely distributed hudson was vs. buildbot..).

I do not think there is any difference regarding distribution.
Both support it fairly well.
My main question mark is regarding structure of OSGeo projects
vs master(s) vs jenkins instance(s)

In buildbot, we followed simple convention that each project has
separate instance (process) with dedicated master.
So, there was clear physical separation, a dedicated URL, separate
status website, etc. I believe it's a good idea to have it like this.

In jenkins, I'm not sure what we can achieve.
I don't want to present developers with single page with all builds of
all projects. It would be messy. Also, it would make it difficult to
maintain privileges.

Looking at jenkins' own build config, I see some categories displayed as
tabs. jenkins documentation is bad, so I could not find anything
regarding this, but perhaps it is at least possible to have projects
displayed on separate tabs like here: Core, Libraries, Plugins, etc.

http://ci.jenkins-ci.org/view/All/

Imagine those tabs are GDAL, GEOS, PROJ.4, etc.

I've planned to describe our idea and expactations and post to jenkins
mailing list [1] asking for suggestion how to organise it. But, family
is visiting me at london, so no time until Monday.
Meanwhile, let's discuss here what we want, as you suggest.

[1] http://groups.google.com/group/jenkinsci-users/topics

BTW, I quickly looked at buildbot. Quite a few new relases since I used
it last time. One of new nice features is console:
http://trac.buildbot.net/wiki/ScreenShots

Still no GUI for configuration though.
Also, some nice programmatic features have been added:
http://buildbot.net/buildbot/docs/current/manual/customization.html

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org

(Note, I'm bumping a pre-historic thread here.)

On 6 October 2011 21:26, Mateusz Loskot <mateusz@loskot.net> wrote:

On 06/10/11 21:08, Frank Warmerdam wrote:

On Thu, Oct 6, 2011 at 1:06 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

On 20/09/11 11:44, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well,
especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

------------------------------------^^^^^^^^^^^^^^^^

Bump.

Mateusz,

I'm concerned that our two servers at OSU OSL are getting
rather crowded with VMs and that it might be hard to
host a big build-and-test VM there right now. I wonder
if John might be able to provide an appropriate VM at
Telascience?

I see. Let's wait and see what's the situation when the dust after
current motions/migrations settles down.

Frank,

Would you have any news about possible machines to use for build automation?

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

Mateusz,

I'd suggest we go ahead and set it up on the buildtest vm at OSL but try to
keep the load somewhat modest for now. We have a budget to buy another
physical machine for OSL this year at which point we could ramp up the
projects on an integration server.

Best regards,
Frank

On Thu, May 17, 2012 at 4:52 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

(Note, I'm bumping a pre-historic thread here.)

On 6 October 2011 21:26, Mateusz Loskot <mateusz@loskot.net> wrote:

On 06/10/11 21:08, Frank Warmerdam wrote:

On Thu, Oct 6, 2011 at 1:06 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

On 20/09/11 11:44, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well,
especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

------------------------------------^^^^^^^^^^^^^^^^

Bump.

Mateusz,

I'm concerned that our two servers at OSU OSL are getting
rather crowded with VMs and that it might be hard to
host a big build-and-test VM there right now. I wonder
if John might be able to provide an appropriate VM at
Telascience?

I see. Let's wait and see what's the situation when the dust after
current motions/migrations settles down.

Frank,

Would you have any news about possible machines to use for build automation?

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

I think a build machine is better suited for telascience (if available)
or Michael mentioned when I saw him in april that he was putting in a
new machine where the WMS shootout boxes are. I'd like to pursue those 1st.

Anyone in a position to take up the conversation with those 2 gate
keepers about the idea?

Thanks,
Alex

On 05/17/2012 04:59 PM, Frank Warmerdam wrote:

Mateusz,

I'd suggest we go ahead and set it up on the buildtest vm at OSL but try to
keep the load somewhat modest for now. We have a budget to buy another
physical machine for OSL this year at which point we could ramp up the
projects on an integration server.

Best regards,
Frank

On Thu, May 17, 2012 at 4:52 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

(Note, I'm bumping a pre-historic thread here.)

On 6 October 2011 21:26, Mateusz Loskot <mateusz@loskot.net> wrote:

On 06/10/11 21:08, Frank Warmerdam wrote:

On Thu, Oct 6, 2011 at 1:06 PM, Mateusz Loskot <mateusz@loskot.net> wrote:

On 20/09/11 11:44, Mateusz Loskot wrote:

Hi,

It's been ridiculously long time since the Buildbot gone offline.
Buildbot as the continues integration support had worked extremely well,
especially for the C camp, before it disappeared.
I want to bring it back.
I discussed this shortly with Howard and Chris in Denver
and we agreed to replace old-good Buildbot with http://jenkins-ci.org

If SAC agrees and if we have hardware resources for this,
I'm ready to start working on it. Where to start?

------------------------------------^^^^^^^^^^^^^^^^

Bump.

Mateusz,

I'm concerned that our two servers at OSU OSL are getting
rather crowded with VMs and that it might be hard to
host a big build-and-test VM there right now. I wonder
if John might be able to provide an appropriate VM at
Telascience?

I see. Let's wait and see what's the situation when the dust after
current motions/migrations settles down.

Frank,

Would you have any news about possible machines to use for build automation?

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

On 18 May 2012 01:09, Alex Mandel <tech_dev@wildintellect.com> wrote:

I think a build machine is better suited for telascience (if available)
or Michael mentioned when I saw him in april that he was putting in a
new machine where the WMS shootout boxes are. I'd like to pursue those 1st.

Anyone in a position to take up the conversation with those 2 gate
keepers about the idea?

Any news regarding this?

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net