Scheduled Maintenance (March 4th 2023)
HBH will be offline for around 2 hours on March 4th at 11am UTC+0.
We are migrating our current database infrastructure to new servers and changing our MySQL proxy to a different system which allows better monitoring of the system and gives us a better idea of MySQL traffic and query health.
This upgrade is also to support version 3.0.0 so we can start to deploy our nightly builds of version 3.0.0 to our infrastructure to allow staff to test changes and tune version 3.0.0. It also allows us to make some large changes to the labs system to help reduce the projected costs of the labs and allow us to test the labs on our network and infrastructure which will run the labs once version 3.0.0 is complete.
@Futility Thanks. I will at some point finish the blog post on the version 2.0.0 infrastructure and the migration from our old infrastructure to the current one.
I have started another blog post about the version 3.0.0 infrastructure and the labs system infrastructure as I love reading about infrastructure and as you know we try to be public with as much possible now :)
We have had to revert the database infrastructure changes and in turn roll back the database itself back to 12:50 today. The new database infrastructure was missing 46% of writes into the cluster and was also dropping 56% of reads on our queries. We also had some issues with the reported number of database reads as the report number was far too high based on our past two years of stats from our monitoring system.
As all of these issues were not found during our testing we are currently bulid a new database cluster and we will keep the old new database cluster so we can do some further debuging to find out what has caused this issues and then compare the stats and numbers to the current production cluster.
Once we are happy with the new cluster we will then do a staged rollout of the database cluster this time, where we will send 20% of read traffic to the new cluster and then monitor the stats and stability of the site. Then we will increase this percentage untill all reads are going to the new cluster, once we are happy with the reads we will then do the same for our writes which will slow the site down a little as it will need to write the queries to two different cluster but then once we are happy with the new cluster we will then migrate to the new database infrastructure.
Apologies for the issues today and the lost of a handfull of writes to the database due to the rollback. With this new plan in place it should ensure that the future migration of the database shouldn’t have any data loss again.
Thank you for the Update @Mordak - I can’t wait for HBH V3 to be released. I’m so glad that HBH is still up, running & maintained. - I bought Merch from here back in the day & I plan to again soon. I’m also looking forward to the new challenges.
Kind Regards,