[SAC] Munin Notification mail.osgeo.org Disk usage in percent

neteler · October 5, 2013, 8:25pm

Hi SAC,

I spent a while yesterday to clean up the mail server disk (again).

Now out of space once more.
http://webextra.osgeo.osuosl.org/munin/osgeo.org/mail.osgeo.org-df.html

One problem is that there are zillion of replicated attachments in
/var/lib/mailman/archives/private/osgeojapan-board/attachments/

which continue to accumulate. While I had limited the attachment size to 200k:

config_list --outputfile - osgeojapan-board |grep size
max_message_size = 200000
admin_member_chunksize = 30
digest_size_threshhold = 30

... the attachments are in the multi MB size.

Gosh, I just realize that the value might be given in kb, not byte.
ok, I'll change that right away.
Done.

config_list --outputfile - osgeojapan-board |grep size
max_message_size = 350
admin_member_chunksize = 30
digest_size_threshhold = 30

I have done some de-duplication in the attachments folder, now we have
3.2 GB free.
So let's see what happens next.

Markus

---------- Forwarded message ----------
From: <munin@webextra.osgeo.osuosl.org>
Date: Sat, Oct 5, 2013 at 10:00 PM
Subject: Munin Notification mail.osgeo.org Disk usage in percent
To: munin@localhost

osgeo.org :: mail.osgeo.org :: Disk usage in percent
CRITICALs: / is 98.04 (outside range [:98]).

Alex_M · October 5, 2013, 8:31pm

On 10/05/2013 01:25 PM, Markus Neteler wrote:

Hi SAC,

I spent a while yesterday to clean up the mail server disk (again).

Now out of space once more.
http://webextra.osgeo.osuosl.org/munin/osgeo.org/mail.osgeo.org-df.html

One problem is that there are zillion of replicated attachments in
/var/lib/mailman/archives/private/osgeojapan-board/attachments/

which continue to accumulate. While I had limited the attachment size to 200k:

config_list --outputfile - osgeojapan-board |grep size
max_message_size = 200000
admin_member_chunksize = 30
digest_size_threshhold = 30

... the attachments are in the multi MB size.

Gosh, I just realize that the value might be given in kb, not byte.
ok, I'll change that right away.
Done.

config_list --outputfile - osgeojapan-board |grep size
max_message_size = 350
admin_member_chunksize = 30
digest_size_threshhold = 30

I have done some de-duplication in the attachments folder, now we have
3.2 GB free.
So let's see what happens next.

Markus

Thanks Markus, I had been wondering why it was growing so fast despite
the previous change. Hopefully that explains it. Has anyone heard back
from the Japan members about why they keep sending such huges files on
their list?

Thanks,
Alex

neteler · October 5, 2013, 9:26pm

I still think that we should flush the mailman queue of undelivered
emails, removing those intented for hotmail:

mail:/var/log# du -hs mail.* | grep G
4.2G mail.info
7.0G mail.info.1
4.2G mail.log
7.0G mail.log.1

Sample (name obscured):
Oct 5 14:06:54 mail postfix/error[30909]: 39BAE354BA4:
to=<XXXXXX@hotmail.co.uk>, relay=none, delay=342277,
delays=342207/70/0/0.33, dsn=4.4.2, status=deferred (delivery
temporarily suspended: lost connection with
mx4.hotmail.com[65.55.37.72] while sending RCPT TO)

mail:/var/log# grep 'mx4.hotmail.com' mail.info | wc -l
2777287

These bouncing emails create an incredible amount of log data.

Markus

Martin_Spott · October 6, 2013, 12:38am

On Sat, Oct 05, 2013 at 11:26:37PM +0200, Markus Neteler wrote:

I still think that we should flush the mailman queue of undelivered
emails, removing those intented for hotmail:

Purging the queue from Hotmail-recipients in process (takes a while),

Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

neteler · October 10, 2013, 6:20am

Hi,

the mail server disk is filling up again.

Open issue: the attachment duplication here:

mail:/var/lib/mailman/archives/private/osgeojapan-board/attachments/20130529/5396c3c3#
l
total 643968
-rw-rw-r-- 1 list list 3827071 Sep 16 18:29 attachment.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 17:34 attachment-0100.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 17:38 attachment-0101.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 21:44 attachment-0102.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 21:51 attachment-0103.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 21:59 attachment-0104.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 22:18 attachment-0105.zip
-rw-rw-r-- 1 list list 3827071 Sep 27 08:43 attachment-0106.zip
-rw-rw-r-- 1 list list 3827071 Sep 27 12:00 attachment-0107.zip
-rw-rw-r-- 1 list list 3827071 Sep 28 12:01 attachment-0108.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 12:01 attachment-0109.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 15:55 attachment-0110.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 17:57 attachment-0111.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 18:03 attachment-0112.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 18:11 attachment-0113.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 18:54 attachment-0114.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 19:55 attachment-0115.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 22:35 attachment-0116.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 02:10 attachment-0117.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 03:12 attachment-0118.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:33 attachment-0119.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:34 attachment-0120.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:37 attachment-0121.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:38 attachment-0122.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 12:00 attachment-0123.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 16:51 attachment-0124.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 17:48 attachment-0125.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 20:04 attachment-0126.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 21:41 attachment-0127.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:01 attachment-0128.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:27 attachment-0129.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:30 attachment-0130.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:35 attachment-0131.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:58 attachment-0132.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 23:20 attachment-0133.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 00:04 attachment-0134.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 00:07 attachment-0135.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 01:42 attachment-0136.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 02:45 attachment-0137.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 03:56 attachment-0138.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 03:57 attachment-0139.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 04:59 attachment-0140.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 05:30 attachment-0141.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 05:42 attachment-0142.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 05:48 attachment-0143.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 06:52 attachment-0144.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 07:09 attachment-0145.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 07:19 attachment-0146.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 07:23 attachment-0147.zip
...
...

Only those attachments are reaching 10GB again!
http://webextra.osgeo.osuosl.org/munin/osgeo.org/mail.osgeo.org-df.html

Markus

Frank_Warmerdam · October 10, 2013, 6:32am

Markus,

I don't know what is happening, but I've put a crontab job to clean
all the osgeo japan board attachments nightly in place.

PS. When bacula is running it really hammers things.

Best regards,
Frank

On Wed, Oct 9, 2013 at 11:20 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

the mail server disk is filling up again.

Open issue: the attachment duplication here:

mail:/var/lib/mailman/archives/private/osgeojapan-board/attachments/20130529/5396c3c3#
l
total 643968
-rw-rw-r-- 1 list list 3827071 Sep 16 18:29 attachment.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 17:34 attachment-0100.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 17:38 attachment-0101.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 21:44 attachment-0102.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 21:51 attachment-0103.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 21:59 attachment-0104.zip
-rw-rw-r-- 1 list list 3827071 Sep 26 22:18 attachment-0105.zip
-rw-rw-r-- 1 list list 3827071 Sep 27 08:43 attachment-0106.zip
-rw-rw-r-- 1 list list 3827071 Sep 27 12:00 attachment-0107.zip
-rw-rw-r-- 1 list list 3827071 Sep 28 12:01 attachment-0108.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 12:01 attachment-0109.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 15:55 attachment-0110.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 17:57 attachment-0111.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 18:03 attachment-0112.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 18:11 attachment-0113.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 18:54 attachment-0114.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 19:55 attachment-0115.zip
-rw-rw-r-- 1 list list 3827071 Sep 29 22:35 attachment-0116.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 02:10 attachment-0117.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 03:12 attachment-0118.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:33 attachment-0119.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:34 attachment-0120.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:37 attachment-0121.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 04:38 attachment-0122.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 12:00 attachment-0123.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 16:51 attachment-0124.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 17:48 attachment-0125.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 20:04 attachment-0126.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 21:41 attachment-0127.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:01 attachment-0128.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:27 attachment-0129.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:30 attachment-0130.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:35 attachment-0131.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 22:58 attachment-0132.zip
-rw-rw-r-- 1 list list 3827071 Sep 30 23:20 attachment-0133.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 00:04 attachment-0134.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 00:07 attachment-0135.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 01:42 attachment-0136.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 02:45 attachment-0137.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 03:56 attachment-0138.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 03:57 attachment-0139.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 04:59 attachment-0140.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 05:30 attachment-0141.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 05:42 attachment-0142.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 05:48 attachment-0143.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 06:52 attachment-0144.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 07:09 attachment-0145.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 07:19 attachment-0146.zip
-rw-rw-r-- 1 list list 3827071 Oct 1 07:23 attachment-0147.zip
...
...

Only those attachments are reaching 10GB again!
http://webextra.osgeo.osuosl.org/munin/osgeo.org/mail.osgeo.org-df.html

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

neteler · October 10, 2013, 10:16am

On Thu, Oct 10, 2013 at 8:32 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Markus,

I don't know what is happening, but I've put a crontab job to clean
all the osgeo japan board attachments nightly in place.

Ok. Maybe the machine needs a reboot?

PS. When bacula is running it really hammers things.

Would be easy to solve by running bacula with a nice level 10 rather
than 0 (as we do on our PGIS cluster).
I have a cronjob for renicing the bacula daemon which I am happy to
donate if there is interest.

Markus