All,
We will be upgrading one of our MySQL clusters (mysql1-vip (mysql1/mysql2)) next Wednesday, April 20th, 2022 between 17:00-20:00 UTC (10am-1pm PDT).
We are upgrading MySQL (Percona) from 5.5 to 5.6. This is the first step towards managing the cluster under Chef and upgrading to MySQL (Percona) 5.7 at a future date. Expect more maintenance on this cluster in the near future.
This will result in up to 20 minutes of downtime in the middle of the window as we fail-over the cluster. We estimate that the actual downtime will be far less than that bound.
We have tested this upgrade in a separate environment and believe that it will have minimal impact to projects on the cluster. We will make a full backup of the cluster before the upgrade work begins.
If you are using the vip1 cluster, please make sure you are using the mysql1-vip.osuosl.org address instead of mysql1/2.osuosl.org directly or additional outages will occur.
If you have any questions or concerns, please let us know. We’ll be in #osuosl on Libera Chat in case any one needs to contact us directly.
Thanks!
–
Ethan Hampton
Student Systems Engineer
Oregon State University | Open Source Lab
All,
We had an outage of the mysql1-vip cluster from 20:18 to 21:04 UTC today (Apr 18th) related to the final backup and preparation for this upgrade. Xrabackup issued a “FLUSH TABLES WITH READ LOCK” for the final stage of the backup process. When combined with a long running query, this exhausted the MySQL connection pool and resulted in a read/write outage.
We will take steps to mitigate this for the upgrade on the 20th. Please let us know if you have any questions or concerns.
Best-
···
On 4/13/22 1:53 PM, Ethan Hampton wrote:
All,
We will be upgrading one of our MySQL clusters (mysql1-vip (mysql1/mysql2)) next Wednesday, April 20th, 2022 between 17:00-20:00 UTC (10am-1pm PDT).
We are upgrading MySQL (Percona) from 5.5 to 5.6. This is the first step towards managing the cluster under Chef and upgrading to MySQL (Percona) 5.7 at a future date. Expect more maintenance on this cluster in the near future.
This will result in up to 20 minutes of downtime in the middle of the window as we fail-over the cluster. We estimate that the actual downtime will be far less than that bound.
We have tested this upgrade in a separate environment and believe that it will have minimal impact to projects on the cluster. We will make a full backup of the cluster before the upgrade work begins.
If you are using the vip1 cluster, please make sure you are using the mysql1-vip.osuosl.org address instead of mysql1/2.osuosl.org directly or additional outages will occur.
If you have any questions or concerns, please let us know. We’ll be in #osuosl on Libera Chat in case any one needs to contact us directly.
Thanks!
–
Ethan Hampton
Student Systems Engineer
Oregon State University | Open Source Lab
All,
Due to Xtrabackup requiring a read lock for part of the backup process, there will be an additional outage at the beginning of the maintenance window so we can complete a final backup before the cluster upgrade. We will do our best to minimize the outage time.
Please let us know if you have any questions or concerns.
···
On 4/18/22 2:41 PM, Ethan Hampton wrote:
All,
We had an outage of the mysql1-vip cluster from 20:18 to 21:04 UTC today (Apr 18th) related to the final backup and preparation for this upgrade. Xrabackup issued a “FLUSH TABLES WITH READ LOCK” for the final stage of the backup process. When combined with a long running query, this exhausted the MySQL connection pool and resulted in a read/write outage.
We will take steps to mitigate this for the upgrade on the 20th. Please let us know if you have any questions or concerns.
Best-
--
Ethan Hampton
Student Systems Engineer
Oregon State University | Open Source Lab
On 4/13/22 1:53 PM, Ethan Hampton wrote:
All,
We will be upgrading one of our MySQL clusters (mysql1-vip (mysql1/mysql2)) next Wednesday, April 20th, 2022 between 17:00-20:00 UTC (10am-1pm PDT).
We are upgrading MySQL (Percona) from 5.5 to 5.6. This is the first step towards managing the cluster under Chef and upgrading to MySQL (Percona) 5.7 at a future date. Expect more maintenance on this cluster in the near future.
This will result in up to 20 minutes of downtime in the middle of the window as we fail-over the cluster. We estimate that the actual downtime will be far less than that bound.
We have tested this upgrade in a separate environment and believe that it will have minimal impact to projects on the cluster. We will make a full backup of the cluster before the upgrade work begins.
If you are using the vip1 cluster, please make sure you are using the mysql1-vip.osuosl.org address instead of mysql1/2.osuosl.org directly or additional outages will occur.
If you have any questions or concerns, please let us know. We’ll be in #osuosl on Libera Chat in case any one needs to contact us directly.
Thanks!
All,
The upgrade to MySQL (Percona) 5.6 on the mysql1-vip cluster has been completed successfully! As expected, we had about 40 minutes of downtime to complete the backup and then about 3 minutes while we failed over the cluster.
Please reach out if you run into any issues. Expect information on additional maintenance in the near future.
Thanks!
···
On 4/13/22 1:53 PM, Ethan Hampton wrote:
All,
We will be upgrading one of our MySQL clusters (mysql1-vip (mysql1/mysql2)) next Wednesday, April 20th, 2022 between 17:00-20:00 UTC (10am-1pm PDT).
We are upgrading MySQL (Percona) from 5.5 to 5.6. This is the first step towards managing the cluster under Chef and upgrading to MySQL (Percona) 5.7 at a future date. Expect more maintenance on this cluster in the near future.
This will result in up to 20 minutes of downtime in the middle of the window as we fail-over the cluster. We estimate that the actual downtime will be far less than that bound.
We have tested this upgrade in a separate environment and believe that it will have minimal impact to projects on the cluster. We will make a full backup of the cluster before the upgrade work begins.
If you are using the vip1 cluster, please make sure you are using the mysql1-vip.osuosl.org address instead of mysql1/2.osuosl.org directly or additional outages will occur.
If you have any questions or concerns, please let us know. We’ll be in #osuosl on Libera Chat in case any one needs to contact us directly.
Thanks!
--
Ethan Hampton
Student Systems Engineer
Oregon State University | Open Source Lab