[SAC] [OSGeo] #2853: tracsvn ran out of disk space

#2853: tracsvn ran out of disk space
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2022-II
Component: Systems Admin | Keywords:
---------------------------+---------------------------------------
Tracsvn ran out of disk space again. I deleted a snapshot to clear space.

I'm going to check if there is something eating up space.

So this broke both gitea and trac since when server runs out of disk
space, PostgreSQL goes into panick mode and shuts itself down to prevent
data corruption. Both use PostgreSQL
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2853&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2853: tracsvn ran out of disk space
---------------------------+----------------------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2022-II
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+----------------------------------------
Changes (by robe):

* status: new => closed
* resolution: => fixed

Comment:

I didn't see anything out of the ordinary. gitea's git folder is taking
up 66GB and /var/www/svn takes up about 25GB. Those are the biggest
eaters of disk space. Followed by /var/log which is about 2.5GB

So I assume it's just churn of data. I looked at the snapshots which the
500GB I had allocated was partly used for. I deleted a couple more
snapshots and upped the capacity from 400G to 700GB. using

{{{
lxc config edit tracsvn
}}}

So now there is 241G available.

Command to check snapshot usage is below for future reference

{{{
  zfs list -t snapshot | grep tracsvn
}}}

The snapshot sizes are averaging anywhere from 2GB - 19 GB and we have a
retention of 30 days.
as shown below

{{{
osgeo7/containers/tracsvn@snapshot-snap937
2.19G - 97.6G -
osgeo7/containers/tracsvn@snapshot-snap938
1.45G - 98.2G -
osgeo7/containers/tracsvn@snapshot-snap939
2.71G - 104G -
osgeo7/containers/tracsvn@snapshot-snap940
5.63G - 108G -
osgeo7/containers/tracsvn@snapshot-snap941
386M - 99.8G -
osgeo7/containers/tracsvn@snapshot-snap942
2.98G - 104G -
osgeo7/containers/tracsvn@snapshot-snap943
9.61G - 115G -
osgeo7/containers/tracsvn@snapshot-snap944
6.83G - 114G -
osgeo7/containers/tracsvn@snapshot-snap945
8.88G - 113G -
osgeo7/containers/tracsvn@snapshot-snap946
14.0G - 121G -
osgeo7/containers/tracsvn@snapshot-snap947
10.0G - 119G -
osgeo7/containers/tracsvn@snapshot-snap948
7.70G - 115G -
osgeo7/containers/tracsvn@snapshot-snap949
10.7G - 119G -
osgeo7/containers/tracsvn@snapshot-snap950
76.1M - 130G -
osgeo7/containers/tracsvn@snapshot-snap951
143M - 134G -
osgeo7/containers/tracsvn@snapshot-snap952
1.76G - 121G -
osgeo7/containers/tracsvn@snapshot-snap953
10.5G - 124G -
osgeo7/containers/tracsvn@snapshot-snap954
6.52G - 123G -
osgeo7/containers/tracsvn@snapshot-snap955
16.3G - 131G -
osgeo7/containers/tracsvn@snapshot-snap956
10.3G - 117G -
osgeo7/containers/tracsvn@snapshot-snap957
24.2G - 121G -
osgeo7/containers/tracsvn@snapshot-snap960
17.5G - 114G -
osgeo7/containers/tracsvn@snapshot-snap961
6.64G - 111G -
osgeo7/containers/tracsvn@snapshot-snap962
7.10G - 119G -
osgeo7/containers/tracsvn@snapshot-snap963
6.42G - 114G -
osgeo7/containers/tracsvn@snapshot-snap964
8.96G - 113G -
osgeo7/containers/tracsvn@snapshot-snap965
6.49G - 111G -
osgeo7/containers/tracsvn@snapshot-snap966
9.31G - 114G -
osgeo7/containers/tracsvn@snapshot-snap967
85.9M - 109G -
osgeo7/containers/tracsvn@snapshot-for-osgeo4
1.21G - 115G -
}}}

So worsed case 30*20 = 600Gb for snapshot and actual container is using
110GB. So we'd need 600 + 110GB = 710GB to handle worsed case scenario.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2853#comment:1&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.