All,
This morning we experienced some power issues in the machine room where most of our servers are hosted. There was a planned building power outage however it was assumed all generator and UPS systems would work properly. Unfortunately for some reason, some of the systems lost power and had to be manually brought back online. I’m working with the data center manager to figure out what the core issue was. In addition, I’m going to be on-site later today to double check power connections to the hosts that did have a problem.
If you have any system or service that is still offline, please let me know by sending an email to support@osuosl.org. I believe we have resolved all of the outstanding issues.
Thanks for your patience.
···
Lance Albertson
Director
Oregon State University | Open Source Lab
All,
I wanted to send an update for the power issues from this morning. I investigated the hosts that lost power and discovered one of them had a bad PSU and lost all power when the other good PSU lost power (I replaced it this morning). There were other hosts that had their power not properly split between the two power feeds which I have since fixed. Other hosts that were affected are racked in a part of the datacenter that only has one power feed.
I’m still waiting to hear why that particular power feed completely lost power instead of the UPS holding power.
Thanks-
···
Lance Albertson
Director
Oregon State University | Open Source Lab
All,
I got the following update from the data center manager which provides the root cause for the issue yesterday:
“I had our UPS maintenance tech in last night and it seems that one of the batteries in the battery string failed and that is what caused it not to hold up on the switch over. I have them on-site this morning replacing the failed battery.”
There is another planned outage tomorrow morning from 5-7am PST but I don’t expect any issues this time around.
···
Lance Albertson
Director
Oregon State University | Open Source Lab