Hi all,
We recently experienced a power event at the State Data Center where our machines are hosted. I am currently waiting for an official update on the cause.
One of our links to LinkOregon went down but has since reconnected, and everything appears to be coming back online. I am currently checking the single PSU hosts to ensure there are no lingering issues. I will also investigate why part of the network went offline, as our infrastructure was designed to handle these situations.
I will send an update as soon as I have more information.
Thanks-
Hi all,
Following up on this morning’s power event, I now have an explanation from Oregon DAS, who manages the State Data Center.
The DAS Facilities team was performing routine building battery maintenance that involved switching power from the A side to the B side. This work was not run through their change management process ahead of time, so we did not receive advance notice, which is why the alerts came as a surprise. DAS has acknowledged this and will run this type of work through change management with a heads-up notice going forward.
We have two uplinks to LinkOregon via an MLAG on their end. During one of the switchovers, one of their two routers went down while the other stayed up. As traffic failed over between the LinkOregon routers and ours, we saw a brief network blip. The failover worked as intended and the downed router has since come back online. We are following up with LinkOregon on why that router dropped during a single-side switchover, since it should have stayed powered.
Impact was minimal. Our infrastructure is designed to ride through A/B switchovers, and I have verified the single-PSU hosts for any lingering issues. Everything else looked fine and we are back to normal.
DAS had additional battery work planned through Friday, but they have called it off for the rest of the week. The remaining work will be rescheduled through proper change management, so we should get advance notice next time and can pass that along to you.
I will follow up if anything further comes out of the LinkOregon investigation.
Thanks-