#2928: tracsvn been down since for 4 hrs out of disk space
---------------------------+--------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Keywords:
---------------------------+--------------------------------------
Looks like trac went down again as it ran out of disk space.
I'm assuming it's the same issue as before with gitea cache going to
check.
I increased it's disk quota to buy some time. Quota was 900GB before and
is now at 1.2TB
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2928>
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#2928: tracsvn been down since for 4 hrs out of disk space
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
Garbage collecting cleared up 100GB but even with 100GB the server was
only taking up about 300GB, so probably mostly resulting from big
snapshots of the server which are retained for 30 days.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2928#comment:1>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2928: tracsvn been down since for 4 hrs out of disk space
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
On closer inspection I think this might be related to our slow network in
backup. I'm going to reboot osgeo4 in a bit to see if it helps.
I saw 4 tracsvn backups in progress and just killed them. Which means it
was probably holding 4 copies of tracsvn (so like 150GB for each). What
puzzles me the the disk space is not reclaimed, I see some temp snapshots
in zfs, but they don't add up the the 800GB some odd that is claimed to be
used, and the instance itself is only taking up 200GB. I'm sure some is
being held in limbo and is in use but not counting towards snapshot space,
but still eating up the alloted space. I plan to reboot osgeo7 this
weekend too to try to clear up some of this.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2928#comment:2>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2928: tracsvn been down since for 4 hrs out of disk space
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
I'm suspecting this issue will be resolved after I reboot osgeo7. There
is some dead zfs space resulting from failed backups to osgeo4 and I think
they are locked. At very least I should be able to delete these orphan
snapshots once I reboot.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2928#comment:3>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.