[SAC] [Hosting] DNS outage (currently temporarily resolved)

Committees SAC mailing list
1

All,

Starting at around 10:30 AM PDT (1730 UTC), all three of our DNS servers stopped responding to DNS requests. We temporarily recovered the services starting at 10:53 AM PDT (1753 UTC). Initial investigated work suggests we ran into this problem [1] due to the release of CentOS 7.7 today which updated the version of bind. When our configuration management tool ran, it automatically upgraded bind which caused this problem. For now we’ve disabled our configuration management on the nodes pending a more permanent solution. We’ll try to minimize the outage impacts when we deploy the permanent solution.

Apologies for the issues.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820056

···

Lance Albertson

Director
Oregon State University | Open Source Lab

2

We have updated our configuration management to account for the changes in Bind and will be deploying the change at 4pm PDT (2300 UTC) today (in just about 30 minutes). Unfortunately, this may cause a momentary outage on a specific DNS server when we push the update. To minimize the impact, we’ll be pushing this update one server at a time and confirming the fix is working properly.

Thanks-

···

Lance Albertson

Director
Oregon State University | Open Source Lab