[SAC] trac/svn downtime

Folks,

The TracSVN VM hosted at osuosl stopped responding somewhere around 11:15am
EDT this morning. Daniel's service down scripted notified me and a few other
admins, and a couple folks in IRC mentioned the problem. I was unable to
reach the VM by ssh and the trac and svn http services were inaccessable.
Services on other VMs on the same physical server were fine.

I wrote in #osuosl asking for a reboot and minute or two later submitted
an email to support at osuosl.org. Ramereth got back to me within a
couple minutes and was able to reboot the VM. In IRC he wrote:

<Ramereth> coming up, doing a fsck
<Ramereth> not sure what happened. the kvm process was gone it seems :confused:

I'd appreciate it if folks could post here (on the sac list) when similar
problems requiring OSU OSL VM intervention are required so we can get a
sense of which services are flakey, and how often.

Genearally things are working well, but I am concerned that important
services like Trac and SVN might remain inaccessable for many hours if
there are no OSGeo admins around to pass the request on to OSU OSL. Note
that pretty much anyone can do so though it would be good to coordinate
on #osgeo to avoid duplication.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

On Mon, Mar 21, 2011 at 11:44:28AM -0400, Frank Warmerdam wrote:

I'd appreciate it if folks could post here (on the sac list) when similar
problems requiring OSU OSL VM intervention are required so we can get a
sense of which services are flakey, and how often.

I think Daniel's automated 'ping'-monitor already does most of the
work, doesn't it ?

Genearally things are working well, but I am concerned that important
services like Trac and SVN might remain inaccessable for many hours if
there are no OSGeo admins around to pass the request on to OSU OSL. Note
that pretty much anyone can do so though it would be good to coordinate
on #osgeo to avoid duplication.

It's probably the same story as ever: In order to motivate people to
take responsibility you should start by respecting their ideas and
expertise.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

Hi SAC,

currently trac/svn are down and also unreachable by ssh.

http://webextra.osgeo.osuosl.org/munin/osgeo.org/trac.osgeo.org.html

I am in IRC chat with crschmidt but we don't precisely know how to
get a reboot of our VM. Since folks seem to be sleeping now
(US night there), we need to do something so we can do these
reboots ourselves.

I have written to support at osuosl.org.

Below a similar story.

Markus

On Mon, Mar 21, 2011 at 4:44 PM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Folks,

The TracSVN VM hosted at osuosl stopped responding somewhere around 11:15am
EDT this morning. Daniel's service down scripted notified me and a few
other
admins, and a couple folks in IRC mentioned the problem. I was unable to
reach the VM by ssh and the trac and svn http services were inaccessable.
Services on other VMs on the same physical server were fine.

I wrote in #osuosl asking for a reboot and minute or two later submitted
an email to support at osuosl.org. Ramereth got back to me within a
couple minutes and was able to reboot the VM. In IRC he wrote:

<Ramereth> coming up, doing a fsck
<Ramereth> not sure what happened. the kvm process was gone it seems :confused:

I'd appreciate it if folks could post here (on the sac list) when similar
problems requiring OSU OSL VM intervention are required so we can get a
sense of which services are flakey, and how often.

Genearally things are working well, but I am concerned that important
services like Trac and SVN might remain inaccessable for many hours if
there are no OSGeo admins around to pass the request on to OSU OSL. Note
that pretty much anyone can do so though it would be good to coordinate
on #osgeo to avoid duplication.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam,
warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac