[SAC] webextra: / is full

Hi SAC,

by chance I discovered:

webextra: /dev/vda3 20G 19G 0 100% /

I suppose that the machine will crash soon.
Maybe it is a bad idea to put /osgeo into the / partition...

Markus

On Sat, Apr 7, 2012 at 11:52 AM, Markus Neteler <neteler@osgeo.org> wrote:

Hi SAC,

by chance I discovered:

webextra: /dev/vda3 20G 19G 0 100% /

I suppose that the machine will crash soon.
Maybe it is a bad idea to put /osgeo into the / partition...

I have done some brute-force:

rm -f vmap0-access.log.30.gz vmap0-access.log.17.gz
vmap0-access.log.14.gz vmap0-access.log.13.gz vmap0-access.log.12.gz
vmap0-access.log.11.gz

Now 2GB are back.
Apache restarted, now also Munin is back.

Added two favico.ico files (using the OSGeo one) to avoid massive
clutter in the apache error log files.

Still more space is needed on that machine.

Markus

On Sat, Apr 07, 2012 at 12:16:41PM +0200, Markus Neteler wrote:

Still more space is needed on that machine.

Sorry, I'm out of service for the rest of the day (family day now and
glider towing this afternoon), but will have a look at it on sunday, if
nobody beats me).

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

I've just issued a reboot on "webextra" to to get the services into a
clean state.

Wrt. VMap0 tile service on "wbextra", I've replicated most (of maybe
even all) of this on the "sphere.telascience.org" machine a couple of
weeks ago, thus if there's the desire to get it off "webextra", it
probably needs just a DNS change and some testing. I was unsure about
what the current plan is.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

BTW, isn't it wonderful how efficient modern filesystems are :wink:

webextra:/osgeo# pwd
/osgeo
webextra:/osgeo# du -hs .
28G .
webextra:/osgeo# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 20G 15G 4.5G 77% /

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

Back to more reasonable stuff, I'm moving the entire "osgeo" directory
into the separate filesystem, I think that'll release some of the
pressure.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

I've moved the /mapdata/ into /osgeo/, map files in
"/osgeo/tiles/vmap0/" are modified accordingly - please test,

  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

On Sun, Apr 08, 2012 at 03:54:01PM +0200, Martin Spott wrote:

I've moved the /mapdata/ into /osgeo/, map files in
"/osgeo/tiles/vmap0/" are modified accordingly - please test,

.... btw. the (now) "/osgeo/mapdata/vmap.map" as well - I really
wonder, which place is referring to this particular file.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

On 04/08/2012 06:01 AM, Martin Spott wrote:

I've just issued a reboot on "webextra" to to get the services into a
clean state.

Wrt. VMap0 tile service on "wbextra", I've replicated most (of maybe
even all) of this on the "sphere.telascience.org" machine a couple of
weeks ago, thus if there's the desire to get it off "webextra", it
probably needs just a DNS change and some testing. I was unsure about
what the current plan is.

Cheers,
  Martin.

I like the sounds of moving it to sphere.telascience
If you've tested it we should just move the dns for it.
I've just been doing forced logrotates on apache whenever it fills but happened to be offline this weekend.

Thanks,
Alex

On Sun, Apr 08, 2012 at 07:51:04PM -0700, Alex Mandel wrote:

On 04/08/2012 06:01 AM, Martin Spott wrote:

>Wrt. VMap0 tile service on "wbextra", I've replicated most (of maybe
>even all) of this on the "sphere.telascience.org" machine a couple of
>weeks ago, thus if there's the desire to get it off "webextra", it
>probably needs just a DNS change and some testing. I was unsure about
>what the current plan is.

I like the sounds of moving it to sphere.telascience
If you've tested it we should just move the dns for it.

I'm having just little knowledge about the various layers the WMS is
supposed to support, therefore it would be beneficial to know about
some prominent test cases. I'm aware of these file map files, are they
still maintained/needed ?

  /osgeo/mapdata/vmap.map
  /osgeo/tiles/vmap0/openlayers.map
  /osgeo/tiles/vmap0/vmap0.map
  /osgeo/tiles/vmap0/vmap_wms_label.map
  /osgeo/tiles/vmap0/vmap_wms.map

I've just been doing forced logrotates on apache whenever it fills
but happened to be offline this weekend.

I hope I made this job easier by changing the weekly check interval
into a daily one in "/etc/logrotate.d/apache2" :wink:

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

On Sat, Apr 7, 2012 at 11:52 AM, Markus Neteler <neteler@osgeo.org> wrote:

Hi SAC,

by chance I discovered:

webextra: /dev/vda3 20G 19G 0 100% /

... while this one is solved (for now), I would like to bring the
script in this ticket to your attention:

http://trac.osgeo.org/osgeo/ticket/159
"We Need Low Disk notifier on osgeo1 and osgeo2"
--> Shell script to watch the disk space

I would suggest to add this on the critical machines and
simply post the warnings to this list (in the respective Mailman
settings we can enable single addresses to go through, the
email can be fetch from a dedicated account with fetchmail
to avoid a spam hole).

Markus

On 04/11/2012 02:12 PM, Markus Neteler wrote:

On Sat, Apr 7, 2012 at 11:52 AM, Markus Neteler<neteler@osgeo.org> wrote:

Hi SAC,

by chance I discovered:

webextra: /dev/vda3 20G 19G 0 100% /

... while this one is solved (for now), I would like to bring the
script in this ticket to your attention:

http://trac.osgeo.org/osgeo/ticket/159
"We Need Low Disk notifier on osgeo1 and osgeo2"
--> Shell script to watch the disk space

I would suggest to add this on the critical machines and
simply post the warnings to this list (in the respective Mailman
settings we can enable single addresses to go through, the
email can be fetch from a dedicated account with fetchmail
to avoid a spam hole).

Markus

From what I've read munin can do email notifications at certain events. If someone can figure out how to do that it would be fairly effective since munin updates every 5 minutes.

Thanks,
Alex

On Thu, Apr 12, 2012 at 5:52 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:

On 04/11/2012 02:12 PM, Markus Neteler wrote:

...

--> Shell script to watch the disk space

From what I've read munin can do email notifications at certain events. If
someone can figure out how to do that it would be fairly effective since
munin updates every 5 minutes.

Right - some random examples:

http://wiki.kartbuilding.net/index.php/Munin_Statistics#Email_Notifications_from_Munin
http://sysadminman.net/blog/2009/e-mail-alerts-from-munin-for-network-bandwidth-usage-253
http://blog.edseek.com/archives/2006/07/13/munin-alert-email-notification/

Markus

On 04/12/2012 01:34 AM, Markus Neteler wrote:

On Thu, Apr 12, 2012 at 5:52 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:

On 04/11/2012 02:12 PM, Markus Neteler wrote:

...

--> Shell script to watch the disk space

From what I've read munin can do email notifications at certain events. If
someone can figure out how to do that it would be fairly effective since
munin updates every 5 minutes.

Right - some random examples:

http://wiki.kartbuilding.net/index.php/Munin_Statistics#Email_Notifications_from_Munin
http://sysadminman.net/blog/2009/e-mail-alerts-from-munin-for-network-bandwidth-usage-253
http://blog.edseek.com/archives/2006/07/13/munin-alert-email-notification/

Markus

Testing is under way. The one thing those pages don't discuss which I've
found is that you need Postfix or another tool for sending mail
configured correctly on the munin server.

Right now in my tests, it works but each warning is it's own email. I
see some instructions on how to pass the warnings to an outside script,
possibly for collating into a single email but I'm not sure how to
handle that.
http://munin-monitoring.org/wiki/HowToContact

I'll try to finish the setup this week, but the question I have is what
email address should we send to? There is a chance that it can generate
a lot of small emails when stuff goes wrong.

Thanks,
Alex