[SAC] OSGeo3 Raid battery replacement

OSL team,

Now that osgeo4 disks and battery are replaced and working any chance we
could to osgeo3's battery tomorrow Friday Apr 24 ~ 1pm PST?

We've noticed the Web vm is having some big i/o issues. Not sure if it's
the raid card, drbd or kvm cache related. One option is to migrate Web
over to osgeo4 for now and see if that changes things. Any other ideas
are welcome.

We'd probably want to migrate over Secure also while osgeo3 is down.
Which might mean we need to turn off something on osgeo4.

Thanks,
Alex
OSGeo System Administration Committee

See replies inline.

On Thu Apr 24 15:47:59 2014, tech@wildintellect.com wrote:

OSL team,

Now that osgeo4 disks and battery are replaced and working any chance we
could to osgeo3's battery tomorrow Friday Apr 24 ~ 1pm PST?

Sounds fine.

We've noticed the Web vm is having some big i/o issues. Not sure if it's
the raid card, drbd or kvm cache related. One option is to migrate Web
over to osgeo4 for now and see if that changes things. Any other ideas
are welcome.

Caching solutions like varnish tend to reduce I/O demands for webapps, but
this is sort of hard to debug without poking around the various VMs etc.

What I can say is that I/O on osgeo3 is dominated by tracsvn, and that
replacing the battery should cause the RAID firmware to go back to writeback mode.

We'd probably want to migrate over Secure also while osgeo3 is down.
Which might mean we need to turn off something on osgeo4.

We can coordinate this over IRC if you desire.

Justin Dugger
OSU Open Source Lab

On 04/25/2014 10:21 AM, Justin Dugger via RT wrote:

See replies inline.

On Thu Apr 24 15:47:59 2014, tech@wildintellect.com wrote:

OSL team,

Now that osgeo4 disks and battery are replaced and working any chance we
could to osgeo3's battery tomorrow Friday Apr 24 ~ 1pm PST?

Sounds fine.

We've noticed the Web vm is having some big i/o issues. Not sure if it's
the raid card, drbd or kvm cache related. One option is to migrate Web
over to osgeo4 for now and see if that changes things. Any other ideas
are welcome.

Caching solutions like varnish tend to reduce I/O demands for webapps, but
this is sort of hard to debug without poking around the various VMs etc.

Traffic is new to this VM, it had been mostly idle before. I have added
a php caching tool but looks like we might need something more. The
thing that has us curious is if drbd performance is slowing both osgeo3
and osgeo4 since the Web VM is one of the only machines configured for drbd.

What I can say is that I/O on osgeo3 is dominated by tracsvn, and that
replacing the battery should cause the RAID firmware to go back to writeback mode.

That is expected.

We'd probably want to migrate over Secure also while osgeo3 is down.
Which might mean we need to turn off something on osgeo4.

We can coordinate this over IRC if you desire.

Yup, I'll be there.

Justin Dugger
OSU Open Source Lab

Thanks,
Alex

On 04/25/2014 10:21 AM, Justin Dugger via RT wrote:

See replies inline.

On Thu Apr 24 15:47:59 2014, tech@wildintellect.com wrote:

OSL team,

Now that osgeo4 disks and battery are replaced and working any chance we
could to osgeo3's battery tomorrow Friday Apr 24 ~ 1pm PST?

Sounds fine.

We've noticed the Web vm is having some big i/o issues. Not sure if it's
the raid card, drbd or kvm cache related. One option is to migrate Web
over to osgeo4 for now and see if that changes things. Any other ideas
are welcome.

Caching solutions like varnish tend to reduce I/O demands for webapps, but
this is sort of hard to debug without poking around the various VMs etc.

Traffic is new to this VM, it had been mostly idle before. I have added
a php caching tool but looks like we might need something more. The
thing that has us curious is if drbd performance is slowing both osgeo3
and osgeo4 since the Web VM is one of the only machines configured for drbd.

What I can say is that I/O on osgeo3 is dominated by tracsvn, and that
replacing the battery should cause the RAID firmware to go back to writeback mode.

That is expected.

We'd probably want to migrate over Secure also while osgeo3 is down.
Which might mean we need to turn off something on osgeo4.

We can coordinate this over IRC if you desire.

Yup, I'll be there.

Justin Dugger
OSU Open Source Lab

Thanks,
Alex

Battery replaced and web VM modified to use plain disk instead of drbd. Resolving ticket.

Justin

On Fri Apr 25 10:31:46 2014, tech@wildintellect.com wrote:

On 04/25/2014 10:21 AM, Justin Dugger via RT wrote:
> See replies inline.
>
> On Thu Apr 24 15:47:59 2014, tech@wildintellect.com wrote:
>> OSL team,
>>
>> Now that osgeo4 disks and battery are replaced and working any
chance we
>> could to osgeo3's battery tomorrow Friday Apr 24 ~ 1pm PST?
>
> Sounds fine.
>
>> We've noticed the Web vm is having some big i/o issues. Not sure if
it's
>> the raid card, drbd or kvm cache related. One option is to migrate
Web
>> over to osgeo4 for now and see if that changes things. Any other
ideas
>> are welcome.
>
> Caching solutions like varnish tend to reduce I/O demands for
webapps, but
> this is sort of hard to debug without poking around the various VMs
etc.
>
Traffic is new to this VM, it had been mostly idle before. I have
added
a php caching tool but looks like we might need something more. The
thing that has us curious is if drbd performance is slowing both
osgeo3
and osgeo4 since the Web VM is one of the only machines configured for
drbd.

> What I can say is that I/O on osgeo3 is dominated by tracsvn, and
that
> replacing the battery should cause the RAID firmware to go back to
writeback mode.
>
That is expected.

>> We'd probably want to migrate over Secure also while osgeo3 is
down.
>> Which might mean we need to turn off something on osgeo4.
>
> We can coordinate this over IRC if you desire.
>
Yup, I'll be there.

> Justin Dugger
> OSU Open Source Lab
>
>
>
Thanks,
Alex