[SAC] [OSGeo] #2449: Setup Grafana and Prometheus for monitoring of servers

Committees SAC mailing list
1

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+--------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Keywords:
---------------------------+--------------------------------------
When I killed off the old osgeo3, I made all the munin-nodes no longer
useful.

I started to setup a new munin -- but couldn't get it to work -- which got
me thinking maybe there is something better anyway.

Long story short -- after accidentally stumbling on some videos about
Matrix, I came across a combination of using Prometheus with Grafana and
it ended up surprisingly easy to install.

I've only installed nodes so far on the newer servers on osgeo3 and
started to do it on osgeo7.

The web interface for the monitoring is at

https://monitoring.osgeo.org

and authentication is via our LDAP.

I'm going to try installing a node exporter on tracsvn next.

Attached is a sample chart in Grafana for nexus container

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

2

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Changes (by robe):

* Attachment "nexus-chart.png" added.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

3

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------

Old description:

When I killed off the old osgeo3, I made all the munin-nodes no longer
useful.

I started to setup a new munin -- but couldn't get it to work -- which
got me thinking maybe there is something better anyway.

Long story short -- after accidentally stumbling on some videos about
Matrix, I came across a combination of using Prometheus with Grafana and
it ended up surprisingly easy to install.

I've only installed nodes so far on the newer servers on osgeo3 and
started to do it on osgeo7.

The web interface for the monitoring is at

https://monitoring.osgeo.org

and authentication is via our LDAP.

I'm going to try installing a node exporter on tracsvn next.

Attached is a sample chart in Grafana for nexus container

New description:

When I killed off the old osgeo3, I made all the munin-nodes no longer
useful.

I started to setup a new munin -- but couldn't get it to work -- which got
me thinking maybe there is something better anyway.

Long story short -- after accidentally stumbling on some videos about
Matrix, I came across a combination of using Prometheus with Grafana and
it ended up surprisingly easy to install.

I've only installed nodes so far on the newer servers on osgeo3 and
started to do it on osgeo7.

The web interface for the monitoring is at

https://monitoring.osgeo.org

and authentication is via our LDAP.

I'm going to try installing a node exporter on tracsvn next.

Attached is a sample chart in Grafana for nexus container

--

Comment (by robe):

[[Image(nexus-chart.png)]]

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449#comment:1&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

4

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------

Old description:

When I killed off the old osgeo3, I made all the munin-nodes no longer
useful.

I started to setup a new munin -- but couldn't get it to work -- which
got me thinking maybe there is something better anyway.

Long story short -- after accidentally stumbling on some videos about
Matrix, I came across a combination of using Prometheus with Grafana and
it ended up surprisingly easy to install.

I've only installed nodes so far on the newer servers on osgeo3 and
started to do it on osgeo7.

The web interface for the monitoring is at

https://monitoring.osgeo.org

and authentication is via our LDAP.

I'm going to try installing a node exporter on tracsvn next.

Attached is a sample chart in Grafana for nexus container

New description:

When I killed off the old osgeo3, I made all the munin-nodes no longer
useful.

I started to setup a new munin -- but couldn't get it to work -- which got
me thinking maybe there is something better anyway.

Long story short -- after accidentally stumbling on some videos about
Matrix, I came across a combination of using Prometheus with Grafana and
it ended up surprisingly easy to install.

I've only installed nodes so far on the newer servers on osgeo3 and
started to do it on osgeo7.

The web interface for the monitoring is at

https://monitor.osgeo.org

and authentication is via our LDAP.

I'm going to try installing a node exporter on tracsvn next.

Attached is a sample chart in Grafana for nexus container

--

Comment (by robe):

sorry type in url - it's
https://monitor.osgeo.org

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449#comment:2&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

5

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------

Comment (by robe):

okay have configured tracsvn. A bit different than the newer OS because
it's not systemd based.

The same binaries work since its all statically linked go.

I have the steps and binaries in this repo. I'll add the systemd steps in
a bit.

https://git.osgeo.org/gitea/sac/prometheus-config

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449#comment:3&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

6

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------

Comment (by strk):

Great to see modern monitoring coming up !
Does grafana support alerting system ? Since you mention matrix... could
grafana send alerts via matrix to #sac:osgeo.or ?

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449#comment:4&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

7

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Changes (by robe):

* milestone: Sysadmin Contract 2021-I => Sysadmin Contract 2020-I

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449#comment:8&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

8

#2449: Setup Grafana and Prometheus for monitoring of servers
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2020-I
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------------------------
Changes (by robe):

* status: new => closed
* resolution: => fixed

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2449#comment:9&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.