[SAC] [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

Committees SAC mailing list
1

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+-------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Keywords:
---------------------------+-------------------
Planet has informed us that they would like to retire build.geoserver.org
(currently hosted on AWS).

The GeoServer? PSC would like to ask explore options with SAC:

- Is it possible to setup Jenkins on an OSGeo virtual machine(s)
- Some care needs to be taken with both resource use and security
- Or is it recommended to take over AWS hosting (will ask for an estimate
of monthly costs)

We do have the jenkins configuration and so on saved in private OSGeo
gitlab repo presently. The server is setup to create and throwaway
workers.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

2

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

Long term it would be good to have windows and macOS jenkins servers (to
assist open source projects in creating signed installers). We should plan
for this needed even if we do not implement right now.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:1&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

3

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

Discussion: It may be appropriate to use Microsoft Azure pipelines with
offer linux, windows and macOS builds. With enough automation that could
be used to sign an installer.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:2&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

4

Dear OSGeo SAC, admins, volunteers and stakeholders

In email today there is a topic and SAC ticket, related to a Geoserver build-server being discontinued by Planet .. This is a great example of a modern situation. Indeed, a suggestion followed to move the same services, plus build services, plus signing services, to yet another Giant Corp's infrastructure-as-a-service.

SAC at OSGeo -- own your own hardware, run your own AUTH, sign your own software

Anyone who has been unjustly treated in business will know what this means. "Business is never fair" heard this?

Owning your Own Hardware is the right move over years, not hours
why? * much harder to be shut down at short notice; * much less expensive over years (not hours);
* platform access decisions, such as authentication (AUTH) are in the hands of the group involved, not a for-profit, de-personal Corporate entity that does not have your interests in their decision making; * legacy software decisions are in the hands of the group involved; "The King Decides" .. a system of charity, leads to awkward situations, or worse, as things change, on the other hand, owning the hardware and internalizing important decisions, leads to autonomy;

Current news about Google Cloud platform .. running a a compute node on their platform means, you use a Linux with Google modifications ! what could go wrong.

Deal with occassional breakins and re-build the server
Build a trust network that is centered on OSGeo, not Giant Corp

.. the right decision over years, not hours

best regards from California --Brian M Hamlin

5

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by wildintellect):

Can someone chime in how this might relate to our previous and ongoing
build infrastructure? strk or robe?

We should make a list of options to pick between. There seems to be many
ways to go, and something on a cloud provider may scale better. I would
like to rename the ticket to need Build, or CI service to highlight it
need not be Jenkins (which I've had issues securing in the past).

Jody can you provide a timeline on which this needs to happen.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:3&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

6

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by strk):

The only CI service currently offered by OSGeo is http://dronie.osgeo.org.

It is a "Drone CI" server and an handful of "agents" either donated
by devs/users or hosted in OSGeo infrastructure. See
https://wiki.osgeo.org/wiki/Dronie

It is currently used by (to my knowledge):

   - PostGIS
   - GEOS
   - librttopo

The server integrates with Gitea, automatically building what's
pushed to a gitea repository (git.osgeo.org/gitea) and cannot be
connected to other sources (each server connects to a single source).

Windows runners are supported, although we don't currently run any:
https://readme.drone.io/runner/docker/overview/

Jenkins is something PostGIS/GEOS also do use, but outside of OSGeo
infrastructure. It may be good to provide it as a shared facility for
different users. I would note that Jenkins is also known to possibly
integrate with Gitea.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:4&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

7

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by robe):

I have been thinking about this a bit. Sadly I don't think a shared
Jenkins would work too well or at least not in my skill set. We could
setup a dedicate VM/Container for Jenkins.

PostGIS does run it's main jenkins in LXD/LXC container but it doesn't use
throw away slaves.

jive, can we discuss on OSGeo SAC irc the needs of GeoServer. Depending
on your bandwidth needs Amazon or some other Cloud provider might be your
best fit and off course the funds would be paid by OSGeo.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:5&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

8

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

GeoCat has volunteered a build server also, going to discuss in tomorrow's
GeoServer meeting.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:6&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

9

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

A shared build server was used previously for geotools, geowebcache,
geoserver, geoscript, geogig, etc...

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:7&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

10

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

I am ready to work on this if a Nexus instance can be made available.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:8&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

11

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by robe):

jive are we talking about nexus here or jenkins or both?

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:9&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

12

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by robe):

jive it just occurred to me that I can take an image of your geotools
jenkins server like I did with the old osgeo servers when I migrated them.
That might save some work -- if you can give me access or let me know
someone to talk to to orchestrate this, that might be the easiest course
of action.

I also have osgeo3 all setup with lxd installed, so we can use that.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:10&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

13

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

The new jenkins server is now available at build.geoserver.org ...

But we are having some kind of problem accessing repo.osgeo.org with
consistent performance and wanted to ask if there is any network
throttling or limiting in place on repo.osgeo.org? A maven build and a
denial of service attack look very similar with bursty traffic patterns.

Our builds are failing after a sustained load on repo.osgeo.org with
connection timeouts:
<pre>
[ERROR] Failed to execute goal on project geoserver:
Could not resolve dependencies for project
org.geoserver:geoserver:pom:2.18-SNAPSHOT:
Could not transfer artifact org.geotools:gt-
opengis:jar:24-20200503.120242-25 from/to osgeo-snapshots
(https://repo.osgeo.org/repository/snapshot/):
Connect to repo.osgeo.org:443 [repo.osgeo.org/140.211.15.6] failed:
Connection timed out (Connection timed out) -> [Help 1]
</pre>

We have:

- Observed this from build.geoserver.org
- Set up a second build on a different cloud provider to verify the
problem
- however building as individuals around the world does not show this
problem
- builds of other downstream projects (geomea) is are not having a problem

Next steps:

- we are going to test a build that does not use repo.osgeo.org in order
to double check this is an osgeo networking issue

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:11&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

14

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

I have also looked at the stats from monitor.osgeo.org and from
repo.osgeo.org and cannot see any problems (but I do not know what I am
looking for).

Chat for this activity is on https://gitter.im/geotools/geotools

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:12&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

15

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

Okay so we ended up where:

a) build job was stuck
b) nexus was not doing anything

And then it got unstuck!

Here’s the chunk of log around where it was downloading the metadata:

{{{
[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ geotools ---
[INFO] Downloading from nexus: https://repo.osgeo.org/repository/geotools-
snapshots/org/geotools/geotools/24-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded from nexus: https://repo.osgeo.org/repository/geotools-
snapshots/org/geotools/geotools/24-SNAPSHOT/maven-metadata.xml (593 B at 0
B/s)
[INFO] Uploading to nexus: https://repo.osgeo.org/repository/geotools-
snapshots/org/geotools/geotools/24-SNAPSHOT/geotools-24-20200505.181440-26.pom
[INFO] Uploaded to nexus: https://repo.osgeo.org/repository/geotools-
snapshots/org/geotools/geotools/24-SNAPSHOT/geotools-24-20200505.181440-26.pom
(86 kB at 40 kB/s)
}}}

(593 B at 0 B/s) really looks like some sort of throttling to me.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:13&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

16

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

Seeing inconsistent timeouts on the geotools doc deploys:

{{{
+ scp -oStrictHostKeyChecking=no index.zip
geotools@docs.geotools.org:/home/geotools/stable
ssh: connect to host docs.geotools.org port 22: Connection timed out
lost connection
}}}

This is a second osgeo server, The interesting part is that they sometimes
work fine (this failure did happen while geotools-master was downloading
maven-metadata, so might be related)

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:14&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

17

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone:
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+--------------------

Comment (by jive):

Update, we are getting wildly different build times with some very long
delays in europe. A wide range of folks are running builds in different
parts of the world to try and figure out of this is a jenkins issue or a
nexus issue.

Examples:

- usa: 07:26 min
- amsterdam: 52:47 min
- amsterdam: 06:57 min

As shown above "something" has happened and performance has now improved,
and we can see that improvement on the jenkins build server jobs which are
presently running at a steady clip.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:15&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

18

#2411: Jenkins build server to replace build.geoserver.org
---------------------------+---------------------
Reporter: jive | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone:
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------
Changes (by jive):

* status: new => closed
* resolution: => fixed

Comment:

Thanks to GeoCat for hosting!

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:16&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.