[SAC] Networking issues to OSUOSL

Committees SAC mailing list
1

Hi,

I'm having serious trouble getting to many of the OSGeo-hosted servers in
OSUOSL, starting at 6:47AM eastern time.

This appears to be affecting many OSUOSL services, including Drupal,
Freenode, and OSUOSL's own website, so I assume that this is a wider
networking event and not specific to our servers, but wanted to report
it so you were aware.

In some cases, it looks like HTTP traffic isn't let through while ssh
traffic is; in other cases, it just looks like sparadic slowness in
connections.

In most cases, it does not seem like small ICMP traffic is affected;
however, larger packet sizes do seem to be affected. With 200 byte
packets, I'm getting:

19 packets transmitted, 9 packets received, 52.6% packet loss
round-trip min/avg/max/stddev = 108.888/111.002/112.109/1.159 ms

With 150 bytes, I'm getting:

17 packets transmitted, 15 packets received, 11.8% packet loss
round-trip min/avg/max/stddev = 111.543/114.007/117.911/1.896 ms

With 500 bytes, I'm getting nothing at all.

We appreciate any response you can offer on recovery + anything
we can do to help the situation.

Best Regards,
Christopher Schmidt

2

For the record, this got an auto-ticket number of [support.osuosl.org #19824] .

-- Chris

On Nov 2, 2011, at 7:02 AM, ext christopher.schmidt@nokia.com wrote:

Hi,

I'm having serious trouble getting to many of the OSGeo-hosted servers in
OSUOSL, starting at 6:47AM eastern time.

This appears to be affecting many OSUOSL services, including Drupal,
Freenode, and OSUOSL's own website, so I assume that this is a wider
networking event and not specific to our servers, but wanted to report
it so you were aware.

In some cases, it looks like HTTP traffic isn't let through while ssh
traffic is; in other cases, it just looks like sparadic slowness in
connections.

In most cases, it does not seem like small ICMP traffic is affected;
however, larger packet sizes do seem to be affected. With 200 byte
packets, I'm getting:

19 packets transmitted, 9 packets received, 52.6% packet loss
round-trip min/avg/max/stddev = 108.888/111.002/112.109/1.159 ms

With 150 bytes, I'm getting:

17 packets transmitted, 15 packets received, 11.8% packet loss
round-trip min/avg/max/stddev = 111.543/114.007/117.911/1.896 ms

With 500 bytes, I'm getting nothing at all.

We appreciate any response you can offer on recovery + anything
we can do to help the situation.

Best Regards,
Christopher Schmidt
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

3

On Wed Nov 02 11:05:48 2011, christopher.schmidt@nokia.com wrote:

Hi,

I'm having serious trouble getting to many of the OSGeo-hosted servers

in

OSUOSL, starting at 6:47AM eastern time.

This appears to be affecting many OSUOSL services, including Drupal,
Freenode, and OSUOSL's own website, so I assume that this is a wider
networking event and not specific to our servers, but wanted to report
it so you were aware.

In some cases, it looks like HTTP traffic isn't let through while ssh
traffic is; in other cases, it just looks like sparadic slowness in
connections.

In most cases, it does not seem like small ICMP traffic is affected;
however, larger packet sizes do seem to be affected. With 200 byte
packets, I'm getting:

19 packets transmitted, 9 packets received, 52.6% packet loss
round-trip min/avg/max/stddev = 108.888/111.002/112.109/1.159 ms

With 150 bytes, I'm getting:

17 packets transmitted, 15 packets received, 11.8% packet loss
round-trip min/avg/max/stddev = 111.543/114.007/117.911/1.896 ms

With 500 bytes, I'm getting nothing at all.

We appreciate any response you can offer on recovery + anything
we can do to help the situation.

Best Regards,
Christopher Schmidt

Our upstream provider NERO experienced a fault on one linecard of a core
router in Portland this morning, 11/2/2011, at around 0340 PDT (1040
UTC). Between 0340 and 0525 PDT (1040-1225 UTC) traffic was impacted
passing to/from the Internet until the problem was isolated and
resolved. At this point the cause appears to have been a software fault.
It appears that this only affected some of the routes to/from the OSL
from my investigation (primarily non-I2 routes seemed to not be
affected). I am waiting on confirmation on this assumption.

All services should be back to normal as of 0525 PDT. I apologize for
the issues this caused earlier this morning.

Thanks!

--
Alan Sherman
Student Systems Administrator
OSU Open Source Lab