[SAC] [OSGeo] #2367: Dronie agents are out of space

#2367: Dronie agents are out of space
---------------------------+---------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Keywords:
---------------------------+---------------------------------------
As per {{{ disk quota exceeded }}} error message, see
https://dronie.osgeo.org/postgis/postgis/736

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+----------------------------------------

Comment (by strk):

The https://wiki.osgeo.org/wiki/Dronie page give instructions to spot
known agents by reading {{{ /var/log/drone.log* }}} but no such file
exists either in container or host server

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:1&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+----------------------------------------

Comment (by strk):

Just a guess, we could be having agents suffering from lack of garbage-
collection in docker engine: https://git.osgeo.org/gitea/sac/drone-agent-
config/issues/3

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:2&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+----------------------------------------

Comment (by strk):

Build {{{ 740 }}} for postgis/postgis had this problem too (in case it's
possible to tell which agent took which build).

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:3&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+----------------------------------------

Comment (by strk):

If I got it right I can see the failing jobs 740 are running with agent
{{{ stage_machine = @h.pcorp.us }}}

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:4&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+----------------------------------------

Comment (by robe):

oh that's my fault. Okay let me shut that one off. didn't know I still
had it on.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:5&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: new
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+----------------------------------------

Comment (by robe):

I've turned that one off now and rerunning the above filaed job. Though
it df -H showed it had around 250 GB space. So maybe its some docker
partition thing not set right on it.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:6&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2367: Dronie agents are out of space
---------------------------+----------------------------------------
Reporter: strk | Owner: sac@…
     Type: task | Status: closed
Priority: critical | Milestone: Sysadmin Contract 2019-II
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+----------------------------------------
Changes (by robe):

* status: new => closed
* resolution: => fixed

Comment:

I've shut this agent off so closing.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2367#comment:7&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.