[SAC] mail.osgeo.org running out of disk space

Committees SAC mailing list
1

Hi,

in case nobody noticed the munin messages:

mail:/home/neteler# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 69G 65G 910M 99% /

Can we increase the partition size?

Markus

2

On Fri, Sep 13, 2013 at 06:07:18PM +0200, Markus Neteler wrote:

mail:/home/neteler# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 69G 65G 910M 99% /

I won't be surprised if it's the Mailman list archive filling the disk.

Can we increase the partition size?

The entire disk is already full. The machine either needs its disk
extended or have another disk added.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

3

On 09/13/2013 09:30 AM, Martin Spott wrote:

On Fri, Sep 13, 2013 at 06:07:18PM +0200, Markus Neteler wrote:

mail:/home/neteler# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 69G 65G 910M 99% /

I won't be surprised if it's the Mailman list archive filling the disk.

Can we increase the partition size?

The entire disk is already full. The machine either needs its disk
extended or have another disk added.

Cheers,
  Martin.

So we added 10G back in February, mail usage seems to have grown quite a
bit since then
http://webextra.osgeo.osuosl.org/munin/osgeo.org/mail.osgeo.org-df.html

Looks like we've actually done that 2x since we created the VM. Is all
of that legit mail and archives? The growth curve seems quite steep and
long term difficult to sustain. Are those archives compressed?

I estimate we still have 250 G of room that can be used to grow on
osgeo4. Also once the new backup machine is online that will free up 500 G.

I'll prepare a request to augment the disk with OSUOSL but we should
look at reducing the rate of growth if it doesn't impact the service.

Thanks,
Alex

4

On 09/13/2013 10:51 AM, Alex Mandel wrote:

On 09/13/2013 09:30 AM, Martin Spott wrote:

On Fri, Sep 13, 2013 at 06:07:18PM +0200, Markus Neteler wrote:

mail:/home/neteler# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 69G 65G 910M 99% /

I won't be surprised if it's the Mailman list archive filling the disk.

Can we increase the partition size?

The entire disk is already full. The machine either needs its disk
extended or have another disk added.

Cheers,
  Martin.

So we added 10G back in February, mail usage seems to have grown quite a
bit since then
http://webextra.osgeo.osuosl.org/munin/osgeo.org/mail.osgeo.org-df.html

Looks like we've actually done that 2x since we created the VM. Is all
of that legit mail and archives? The growth curve seems quite steep and
long term difficult to sustain. Are those archives compressed?

I estimate we still have 250 G of room that can be used to grow on
osgeo4. Also once the new backup machine is online that will free up 500 G.

I'll prepare a request to augment the disk with OSUOSL but we should
look at reducing the rate of growth if it doesn't impact the service.

Thanks,
Alex

du -ch --max-depth=1 /var
4.0K /var/opt
16K /var/list
4.0K /var/tmp
4.0K /var/local
8.2G /var/log
100M /var/cache
120K /var/run
35M /var/mail
24K /var/lock
116M /var/www
55G /var/lib
127M /var/spool
4.9M /var/backups
63G /var
63G total

Mail stuff from /var/log
-rw-r----- 1 root adm 45K Sep 13 11:21 mail.err
-rw-r----- 1 root adm 7.1K Sep 6 02:40 mail.err.1
-rw-r----- 1 root adm 915 Sep 1 06:25 mail.err.2.gz
-rw-r----- 1 root adm 993 Aug 23 20:10 mail.err.3.gz
-rw-r----- 1 root adm 937 Aug 16 13:52 mail.err.4.gz
-rw-r----- 1 root adm 1.7G Sep 13 11:47 mail.info
-rw-r----- 1 root adm 1.3G Sep 8 06:27 mail.info.1
-rw-r----- 1 root adm 87M Sep 1 06:30 mail.info.2.gz
-rw-r----- 1 root adm 110M Aug 25 06:25 mail.info.3.gz
-rw-r----- 1 root adm 120M Aug 18 06:28 mail.info.4.gz
-rw-r----- 1 root adm 1.7G Sep 13 11:47 mail.log
-rw-r----- 1 root adm 1.3G Sep 8 06:27 mail.log.1
-rw-r----- 1 root adm 87M Sep 1 06:30 mail.log.2.gz
-rw-r----- 1 root adm 110M Aug 25 06:25 mail.log.3.gz
-rw-r----- 1 root adm 120M Aug 18 06:28 mail.log.4.gz

du -ch --max-depth=1 /var/lib/mailman
110M /var/lib/mailman/data
3.0M /var/lib/mailman/qfiles
4.0K /var/lib/mailman/spam
54G /var/lib/mailman/archives <----
25M /var/lib/mailman/messages
142M /var/lib/mailman/lists
55G /var/lib/mailman
55G total

more specifically /var/lib/mailman/archives/private/
Some of the older lists are in the 500-900 MB range.

Is there a way to compress these mbox files?

Thanks,
Alex

5

On Fri, Sep 13, 2013 at 9:10 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:
...

du -ch --max-depth=1 /var

...

8.2G /var/log

...

55G /var/lib

...

du -ch --max-depth=1 /var/lib/mailman
110M /var/lib/mailman/data
3.0M /var/lib/mailman/qfiles
4.0K /var/lib/mailman/spam
54G /var/lib/mailman/archives <----
25M /var/lib/mailman/messages
142M /var/lib/mailman/lists
55G /var/lib/mailman
55G total

more specifically /var/lib/mailman/archives/private/
Some of the older lists are in the 500-900 MB range.

The email increase is not a real surprise since we are now at 200 lists.

Is there a way to compress these mbox files?

I searched around, apparently not.

Markus

6

Folks,

I have scrubbed some older mailman logs to make a bit of space so we have a few days to cope with this situation.

I’m inclined to just allocate more space. The lists server and it’s archive is a valuable service, and it is not immediately clear that it is out of hand though it is accumulating pretty fast.

Best regards,
Frank

···

On Fri, Sep 13, 2013 at 12:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Fri, Sep 13, 2013 at 9:10 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:
…

du -ch --max-depth=1 /var
…
8.2G /var/log
…
55G /var/lib
…

du -ch --max-depth=1 /var/lib/mailman
110M /var/lib/mailman/data
3.0M /var/lib/mailman/qfiles
4.0K /var/lib/mailman/spam
54G /var/lib/mailman/archives <----
25M /var/lib/mailman/messages
142M /var/lib/mailman/lists
55G /var/lib/mailman
55G total

more specifically /var/lib/mailman/archives/private/
Some of the older lists are in the 500-900 MB range.

The email increase is not a real surprise since we are now at 200 lists.

Is there a way to compress these mbox files?

I searched around, apparently not.

Markus

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

7

(cc japan.osgeo at gmail.com to notify their mailman administrator)

On Sat, Sep 14, 2013 at 2:09 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Folks,

I have scrubbed some older mailman logs to make a bit of space so we have a
few days to cope with this situation.

I'm inclined to just allocate more space. The lists server and it's archive
is a valuable service, and it is not immediately clear that it is out of
hand though it is accumulating pretty fast.

Here the Top 20 usages of the OSGeo mailman lists (in MB):

303 postgis-devel
316 mapbender_commits.mbox
317 grass-user.mbox
324 portugal
372 mapserver-users.mbox
410 qgis-user
418 mapguide-commits
434 qgis-developer
487 openlayers-users
579 gdal-dev
736 mapguide-users
762 mapbender_commits
765 postgis-users.mbox
789 grass-dev
850 fdo-commits
883 grass-user
1259 mapserver-users
1860 postgis-users
1968 grass-commit
28885 osgeojapan-board

There are actually huge attachments therein:
29G /var/lib/mailman/archives/private/osgeojapan-board/attachments

The attachment of the "osgeojapan-board" list is unlimited at time:

config_list --outputfile - osgeojapan-board | grep size
max_message_size = 0

Suggestion: restrict the attachment size to 200k in the
"osgeojapan-board" list to avoid that the mail.osgeo.org server disk
collapses.

Markus

8

Markus,

Good catch!

I don’t really understand why the attachments in the osgeojapan-board/attachments are so huge when the osgeojapan-board.mbox file is only 250MB. I suspect something when crazy in processing for the archives. If I don’t hear back from the OSGeo Japan board in a few days, I’ll archive the attachments somewhere and clean them off lists.osgeo.org. I brief poke into them suggested some of it is slide decks. Clearing 30GB of these attachments from the one list will restore the server to only half full.

Alex, based on that I think we could likely retract the request for server growth.

Best regards,

Frank

···

On Sat, Sep 14, 2013 at 2:13 AM, Markus Neteler <neteler@osgeo.org> wrote:

(cc japan.osgeo at gmail.com to notify their mailman administrator)

On Sat, Sep 14, 2013 at 2:09 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Folks,

I have scrubbed some older mailman logs to make a bit of space so we have a
few days to cope with this situation.

I’m inclined to just allocate more space. The lists server and it’s archive
is a valuable service, and it is not immediately clear that it is out of
hand though it is accumulating pretty fast.

Here the Top 20 usages of the OSGeo mailman lists (in MB):

303 postgis-devel
316 mapbender_commits.mbox
317 grass-user.mbox
324 portugal
372 mapserver-users.mbox
410 qgis-user
418 mapguide-commits
434 qgis-developer
487 openlayers-users
579 gdal-dev
736 mapguide-users
762 mapbender_commits
765 postgis-users.mbox
789 grass-dev
850 fdo-commits
883 grass-user
1259 mapserver-users
1860 postgis-users
1968 grass-commit
28885 osgeojapan-board

There are actually huge attachments therein:
29G /var/lib/mailman/archives/private/osgeojapan-board/attachments

The attachment of the “osgeojapan-board” list is unlimited at time:

config_list --outputfile - osgeojapan-board | grep size
max_message_size = 0

Suggestion: restrict the attachment size to 200k in the
“osgeojapan-board” list to avoid that the mail.osgeo.org server disk
collapses.

Markus

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

9

On 09/14/2013 07:01 AM, Frank Warmerdam wrote:

Markus,

Good catch!

I don't really understand why the attachments in the
osgeojapan-board/attachments are so huge when the osgeojapan-board.mbox
file is only 250MB. I suspect something when crazy in processing for the
archives. If I don't hear back from the OSGeo Japan board in a few days,
I'll archive the attachments somewhere and clean them off lists.osgeo.org.
I brief poke into them suggested some of it is slide decks. Clearing 30GB
of these attachments from the one list will restore the server to only half
full.

Alex, based on that I think we could likely retract the request for server
growth.

Best regards,
Frank

On Sat, Sep 14, 2013 at 2:13 AM, Markus Neteler <neteler@osgeo.org> wrote:

(cc japan.osgeo at gmail.com to notify their mailman administrator)

On Sat, Sep 14, 2013 at 2:09 AM, Frank Warmerdam <warmerdam@pobox.com>
wrote:

Folks,

I have scrubbed some older mailman logs to make a bit of space so we

have a

few days to cope with this situation.

I'm inclined to just allocate more space. The lists server and it's

archive

is a valuable service, and it is not immediately clear that it is out of
hand though it is accumulating pretty fast.

Here the Top 20 usages of the OSGeo mailman lists (in MB):

303 postgis-devel
316 mapbender_commits.mbox
317 grass-user.mbox
324 portugal
372 mapserver-users.mbox
410 qgis-user
418 mapguide-commits
434 qgis-developer
487 openlayers-users
579 gdal-dev
736 mapguide-users
762 mapbender_commits
765 postgis-users.mbox
789 grass-dev
850 fdo-commits
883 grass-user
1259 mapserver-users
1860 postgis-users
1968 grass-commit
28885 osgeojapan-board

There are actually huge attachments therein:
29G /var/lib/mailman/archives/private/osgeojapan-board/attachments

The attachment of the "osgeojapan-board" list is unlimited at time:

config_list --outputfile - osgeojapan-board | grep size
max_message_size = 0

Suggestion: restrict the attachment size to 200k in the
"osgeojapan-board" list to avoid that the mail.osgeo.org server disk
collapses.

Markus

As long as we can solve this week I will hold off on the disk space
increase.

Thanks,
Alex

10

Folks,

OK, I haven’t heard back from japan.osgeo@gmail.com so I have scrubbed the OSGeo Japan Board list attachments archive. The emails themselves will presumably be fine, but any attachments will lead to 404s. If critical I imagine we could recover them from backups but packing them up and moving them somewhere else was taking prohibitively long.

As suggested by Markus, I have also tried setting the max message size to 200K for the list. I think that is in place properly. We now have lots of space:

mail:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 69G 38G 28G 58% /
tmpfs 2.0G 0 2.0G 0% /lib/init/rw
udev 2.0G 112K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/vda1 99M 25M 70M 26% /boot

Best regards,
Frank

···

On Sat, Sep 14, 2013 at 7:01 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Markus,

Good catch!

I don’t really understand why the attachments in the osgeojapan-board/attachments are so huge when the osgeojapan-board.mbox file is only 250MB. I suspect something when crazy in processing for the archives. If I don’t hear back from the OSGeo Japan board in a few days, I’ll archive the attachments somewhere and clean them off lists.osgeo.org. I brief poke into them suggested some of it is slide decks. Clearing 30GB of these attachments from the one list will restore the server to only half full.

Alex, based on that I think we could likely retract the request for server growth.

Best regards,

Frank

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

On Sat, Sep 14, 2013 at 2:13 AM, Markus Neteler <neteler@osgeo.org> wrote:

(cc japan.osgeo at gmail.com to notify their mailman administrator)

On Sat, Sep 14, 2013 at 2:09 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Folks,

I have scrubbed some older mailman logs to make a bit of space so we have a
few days to cope with this situation.

I’m inclined to just allocate more space. The lists server and it’s archive
is a valuable service, and it is not immediately clear that it is out of
hand though it is accumulating pretty fast.

Here the Top 20 usages of the OSGeo mailman lists (in MB):

303 postgis-devel
316 mapbender_commits.mbox
317 grass-user.mbox
324 portugal
372 mapserver-users.mbox
410 qgis-user
418 mapguide-commits
434 qgis-developer
487 openlayers-users
579 gdal-dev
736 mapguide-users
762 mapbender_commits
765 postgis-users.mbox
789 grass-dev
850 fdo-commits
883 grass-user
1259 mapserver-users
1860 postgis-users
1968 grass-commit
28885 osgeojapan-board

There are actually huge attachments therein:
29G /var/lib/mailman/archives/private/osgeojapan-board/attachments

The attachment of the “osgeojapan-board” list is unlimited at time:

config_list --outputfile - osgeojapan-board | grep size
max_message_size = 0

Suggestion: restrict the attachment size to 200k in the
“osgeojapan-board” list to avoid that the mail.osgeo.org server disk
collapses.

Markus

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

11

On Mon, Sep 16, 2013 at 12:21:31PM -0700, Frank Warmerdam wrote:

[...] If critical I imagine we could recover them from backups [...]

Note that our backup is quite 'volatile', to put it mildly, there's
almost no history.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------