At approximately 10:12 AM PDT (1712 UTC) mysql3 (the primary master server for our vip-mysql2.osuosl.org cluster) was accidentally rebooted and caused several outages includes services related to both of our OpenStack clusters. No VMs were offline during the outage, however any API queries would have likely failed. All services should have been restored by around 10:24 AM PDT (1724 UTC).
We’ve been working on getting this cluster into a master-master setup so that this would be prevented in the future.
Thanks and sorry for the issues.
–
Lance Albertson
Director
Oregon State University | Open Source Lab