[SAC] build server

Committees SAC mailing list
1

Hi SAC,

thanks for the cpu and ram on the QGIS server :slight_smile:

But we still fall short in disk (to do our nightly builds etc).

From an older mail:

"Falls in line with a build server that has been discussed recently with
postgis team et al (M. Smith offered one)"

What is the status for that one?

I asked Juergen/jef, and he is fine to move these builds to another
server (hoping/willing to have 'cowbuilder' on that???).

So, can anybody shine a light about this machine?

Is it already available? If so can we have credentials for it?

If not available yet? Any road map for it?

Need help with something?

Regards,

Richard Duivenvoorde

2

On 09/29/2013 04:49 AM, Richard Duivenvoorde wrote:

Hi SAC,

thanks for the cpu and ram on the QGIS server :slight_smile:

But we still fall short in disk (to do our nightly builds etc).

From an older mail:

"Falls in line with a build server that has been discussed recently with
postgis team et al (M. Smith offered one)"

What is the status for that one?

I asked Juergen/jef, and he is fine to move these builds to another
server (hoping/willing to have 'cowbuilder' on that???).

So, can anybody shine a light about this machine?

Is it already available? If so can we have credentials for it?

If not available yet? Any road map for it?

Need help with something?

Regards,

Richard Duivenvoorde

We do have disk space that can be added. I was hesitant to do it this
time because the downtime would be longer for qgis which is quite
heavily visited at the moment.

Also I had an idea, that we should take the "Backup" vm and turn at
least part of it into a "Build" vm while we discuss other options to
come online. Step 1 in this process is to move the backups off osgeo4
onto the new osgeo5 hardware. I've been told osgeo5 was racked last week
but not turned on yet. So I expect it to come online this week. At that
point we could slice off some disk or just rename "Backup" to "Build or
Download2" and get started using it for other things.

Thanks,
Alex

3

Hi Alex,

On Sun, Sep 29, 2013 at 09:01:46AM -0700, Alex Mandel wrote:

Also I had an idea, that we should take the "Backup" vm and turn at
least part of it into a "Build" vm [...]

I'm sorry to disagree.

As far as I understand, the OSGeo infrastructure is meant to host the
public 'interface' for OSGeo projects. Quite a few of the sites are
serving as a flagship for the respective projects and some of them have
already proven to be 'slightly' ressource-hungry.

Project build services I've seen in the OSGeo- as well as other
projects' milieus are typically characerized as being hammered to
death, I/O-wise. I don't think we want to run this sort of 'service'
on a precious ressource which the OSGeo server infrastructure is,
because it finally would hurt OSGeo projects' 'production' sites.

And, last but not least, from past exprience I don't trust any promises
about users behaving cooperatively wrt. using free ressources.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

4

On 09/29/2013 09:26 AM, Martin Spott wrote:

Hi Alex,

On Sun, Sep 29, 2013 at 09:01:46AM -0700, Alex Mandel wrote:

Also I had an idea, that we should take the "Backup" vm and turn at
least part of it into a "Build" vm [...]

I'm sorry to disagree.

As far as I understand, the OSGeo infrastructure is meant to host the
public 'interface' for OSGeo projects. Quite a few of the sites are
serving as a flagship for the respective projects and some of them have
already proven to be 'slightly' ressource-hungry.

Project build services I've seen in the OSGeo- as well as other
projects' milieus are typically characerized as being hammered to
death, I/O-wise. I don't think we want to run this sort of 'service'
on a precious ressource which the OSGeo server infrastructure is,
because it finally would hurt OSGeo projects' 'production' sites.

And, last but not least, from past exprience I don't trust any promises
about users behaving cooperatively wrt. using free ressources.

Cheers,
  Martin.

Well QGIS already has been running all these builds on their VM, with
nice. The biggest block has been the amount of disk space, partly
because we didn't know ahead of time that their VM would be doing that
work. In some sense it's actually taking advantage of under-utilized cpu
resources, but I agree there is a threat to I/O.

I agree long term we can't host a lot of building without others
bringing build slaves online, which has been the discussion lately
(postgis channel talked a lot about this idea a few weeks back). QGIS
does have a jenkins service elsewhere, and I wonder if that could play a
role in this.

I'd say the interesting twist is the QGIS builds are all deb package
builds. I've briefly discussed with Jurgen possibly moving the
non-nightlies to launchpad, but that only works for ubuntu and not
debian. As for the nightlies, launchpad seems to slow to keep up with
that. I'm open to some other suggestions.

What QGIS is looking for right now is actually website doc building with
Sphinx, and I suggested if was can move the deb building off their VM
that would enable their website building to happen where it should.
Question is where to move the deb packaging?

Thanks,
Alex

5

On Sun, Sep 29, 2013 at 10:49:02AM -0700, Alex Mandel wrote:

Well QGIS already has been running all these builds on their VM, with
nice.

The KVM infrastructure as a whole doesn't benefit from stuff running
with 'nice' on one of the VM's.
In other words: I observe the QGIS VM doing a lot of writes - the
entire VM hosting just one single !! project - while the Projects VM,
which is supposed to reside on the same KVM host and hosts half a
dozend projects (or even more) is suffering from considerable I/O
latency at the same time.

Not only are QGIS receiving preferred treatment by having their own VM,
no, at the same time they also hurt other projects by consuming more
ressources than half a dozend projects consume in total - and I'm
pretty certain that in the earlier days of this infrastructure there
was general agreement not to allow build services.

Don't get me wrong, I do *not* generally dislike the QGIS folks or
their project, but when using a common ressource pool, everybody should
comply with the same rules.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

6

On 09/29/2013 11:44 AM, Martin Spott wrote:

On Sun, Sep 29, 2013 at 10:49:02AM -0700, Alex Mandel wrote:

Well QGIS already has been running all these builds on their VM, with
nice.

The KVM infrastructure as a whole doesn't benefit from stuff running
with 'nice' on one of the VM's.
In other words: I observe the QGIS VM doing a lot of writes - the
entire VM hosting just one single !! project - while the Projects VM,
which is supposed to reside on the same KVM host and hosts half a
dozend projects (or even more) is suffering from considerable I/O
latency at the same time.

Good Point. I hadn't noticed because no one complained.

Not only are QGIS receiving preferred treatment by having their own VM,
no, at the same time they also hurt other projects by consuming more
ressources than half a dozend projects consume in total - and I'm
pretty certain that in the earlier days of this infrastructure there
was general agreement not to allow build services.

Yes, I believe we never intended to host Code/App building on the
current machines.

Don't get me wrong, I do *not* generally dislike the QGIS folks or
their project, but when using a common ressource pool, everybody should
comply with the same rules.

Cheers,
  Martin.

I think building their docs is fairly appropriate. I'll note QGIS did
get it's own VM, 1. because they asked, 2. because we thought by
separating it we could provide better uptime to other Projects (which
has worked out with regards to apache/ram usage), QGIS issues beyond I/O
have not brought other Projects to a halt.

Long term seems Projects are now asking for help organizing some build
resources. So the question is what can be proposed (QGIS has not ruled
out buying hardware). I've heard others suggest there is hardware
available in other locations, and a build master-slave config could happen.

Yes, I too would like to shift QGIS builds, but I haven't wanted to kick
them off without finding a suitable replacement. Launchpad is out
because it only does Ubuntu and not Debian, and can't be relied on for
nightly builds. There is a preference to keep it all in once place, and
I do see the hosting of the actual debs for download on osgeo servers
100% inline with expected usage.

So I guess I'm fishing for possible solutions, hopefully others can
chime in with options.

Thanks,
Alex

7

Moin Martin.

On Sun, 29. Sep 2013 at 20:44:48 +0200, Martin Spott wrote:

> Well QGIS already has been running all these builds on their VM, with nice.

The KVM infrastructure as a whole doesn't benefit from stuff running with
'nice' on one of the VM's.

Obvious. But that wasn't the intention - just to leave resources for other
services in the same VM.

no, at the same time they also hurt other projects by consuming more
ressources than half a dozend projects consume in total - and I'm pretty
certain that in the earlier days of this infrastructure there was general
agreement not to allow build services.

Oh, sorry. Nobody complained so far and I added the fact that we're running
builds on that VM to the sac page almost three years ago. So I thought it was
ok. Are there more dos'n'donts that we should be aware of?

Anyway, I've moved the builds to one of our company's servers in order get them
to finish. Originally that was just in preparation for the new server, we were
going to rent.

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de
QGIS PSC member (RM) IRC: jef on FreeNode

--
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502

8

Hi Juergen !

On Sun, Sep 29, 2013 at 09:52:00PM +0200, Jürgen E. Fischer wrote:

On Sun, 29. Sep 2013 at 20:44:48 +0200, Martin Spott wrote:

> no, at the same time they also hurt other projects by consuming more
> ressources than half a dozend projects consume in total - and I'm pretty
> certain that in the earlier days of this infrastructure there was general
> agreement not to allow build services.

Oh, sorry. Nobody complained so far and I added the fact that we're running
builds on that VM to the sac page almost three years ago. So I thought it was
ok. Are there more dos'n'donts that we should be aware of?

Well, I don't know who was involved in setting up a per-project VM,
personally I think the idea wasn't very clever in general - at least in
terms of getting along nicely on a shared, common ressource.
As far as I can tell, the Projects VM suffering from I/O latency has
been a varying but persistent issue, people got used to it .... :-/

I have to admit that I was unaware of the QGIS builds, therefore I
never had a closer look at what's going on at the QGIS VM when I/O
issues were on-topic.
If I remember correctly the general rule was to run on the OSGeo
infrastructure what relates directly to the respective web presence -
thus autogen documentation would be ok. because it 'lives' on-site -
but to move all sorts ressource-hungry build/render/custom stuff
off-site. Please object if you think I'm wrong.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------