All,
We will be performing an upgrade on one of our MySQL clusters (mysql-vip2 (mysql3/mysql4)) next Monday, August 23, 2021 between 2-4PM PDT.
date -d '1400 PDT ← Display this time in your local time zone.
Services Impacted:
- OpenStack control plane and API endpoints (all clusters)
- OSL RT
- SQLGrey on OSL mail relays
- Various projects which use this cluster
Summary:
We are upgrading from MySQL (Percona Server) from 5.6 to 5.7. This will bring us to a version which includes additional features that have been required with some projects.
Outage summary:
We have tested the upgrade in a variety of ways (thanks to our student Robert Detjens for doing this hard work!) and believe our migration should have minimal impacts to services. However, there will likely be a small outage when we run mysql_upgrade since it requires a write lock.
Migration steps:
Here’s a summary of the steps we’re planning on taking for the migration:
- Stop configuration management cron job
- Stop replication on both nodes
- Perform upgrade on mysql4 (does not have the VIP)
- Start daemon on mysql4 (will join as secondary), wait for catch up
- Switch VIP to mysql4
- Stop replication again on both nodes
- Perform upgrade on mysql3
- Start daemon on mysql3 (will join as secondary), wait for catch up
- Switch VIP to mysql3
- Run mysql_upgrade on mysql3 (needs write lock)
- Run mysql_upgrade on mysql4 (needs write lock)
- Run and re-enable configuration management
If you have any questions or concerns, please let us know. We’ll be in #osuosl on Libera Chat in case any one needs to run us directly.
We plan on doing a similar upgrade to our other MySQL cluster, but that will happen later and will require additional steps.
Thanks!
···
Lance Albertson
Director
Oregon State University | Open Source Lab
The upgrade has been completed, but there is an issue with SSL connections. The new version is using self-signed certificates which is causing problems for some clients. We are preparing a fix and will have that rolled out shortly.
···
Lance Albertson
Director
Oregon State University | Open Source Lab
Updated certificates have been deployed and clients should now be able to use SSL connections correctly. If there are any further issues, please open a support ticket or reach us in #osuosl on Libera Chat.
On Mon, Aug 23, 2021 at 4:24 PM Lance Albertson <lance@osuosl.org> wrote:
The upgrade has been completed, but there is an issue with SSL connections. The new version is using self-signed certificates which is causing problems for some clients. We are preparing a fix and will have that rolled out shortly.
On Thu, Aug 19, 2021 at 1:25 PM Lance Albertson <lance@osuosl.org> wrote:
All,
We will be performing an upgrade on one of our MySQL clusters (mysql-vip2 (mysql3/mysql4)) next Monday, August 23, 2021 between 2-4PM PDT.
date -d '1400 PDT ← Display this time in your local time zone.
Services Impacted:
- OpenStack control plane and API endpoints (all clusters)
- OSL RT
- SQLGrey on OSL mail relays
- Various projects which use this cluster
Summary:
We are upgrading from MySQL (Percona Server) from 5.6 to 5.7. This will bring us to a version which includes additional features that have been required with some projects.
Outage summary:
We have tested the upgrade in a variety of ways (thanks to our student Robert Detjens for doing this hard work!) and believe our migration should have minimal impacts to services. However, there will likely be a small outage when we run mysql_upgrade since it requires a write lock.
Migration steps:
Here’s a summary of the steps we’re planning on taking for the migration:
- Stop configuration management cron job
- Stop replication on both nodes
- Perform upgrade on mysql4 (does not have the VIP)
- Start daemon on mysql4 (will join as secondary), wait for catch up
- Switch VIP to mysql4
- Stop replication again on both nodes
- Perform upgrade on mysql3
- Start daemon on mysql3 (will join as secondary), wait for catch up
- Switch VIP to mysql3
- Run mysql_upgrade on mysql3 (needs write lock)
- Run mysql_upgrade on mysql4 (needs write lock)
- Run and re-enable configuration management
If you have any questions or concerns, please let us know. We’ll be in #osuosl on Libera Chat in case any one needs to run us directly.
We plan on doing a similar upgrade to our other MySQL cluster, but that will happen later and will require additional steps.
Thanks!
–
Lance Albertson
Director
Oregon State University | Open Source Lab
–
Lance Albertson
Director
Oregon State University | Open Source Lab
Hosting mailing list
Hosting@osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting