Additionally, we encountered an unrelated issue with a small number of sites on our database cluster using a larger than normal number of long-running database queries, in turn leading to database performance issues more widely across Onyx.
For the LiteSpeed issues, we rolled back to an earlier version to resolve the issues whilst we work with LiteSpeed to resolve this issue in the latest version of the software.
The database issues have been tracked down to a faulty plugin on the sites concerned, and we have worked with the clients to remove this and find alternatives whilst the plugin vendor patches the issue. Additionally, we have taken the opportunity to boost database capacity by bringing online additional clusters, and put the sites with the busiest queries on their own cluster to minimise the risk of this recurring. Our team are also developing additional layers of caching to enhance object and query performance.
28th May 2020
No incidents reported
27th May 2020
Database performance issues
We're aware of an issue with reduced database performance and are working on this now. We'll provide an update shortly.
Edit: 18:49 - Performance has returned to normal levels and we're continuing to monitor. Thanks for your patience.
Slow database queries
We're aware of there being an issue with slow queries on the database cluster, we hope to have this resolved shortly.
Edfit 15:09: This is now resolved and we're continuing to monitor. We're in the process of implementing a long-term fix for this now and apologise for any inconvenience caused.
26th May 2020
Caching is causing sites to not load correctly
We're aware of an issue where our caching layer is causing some elements of a site to not load, showing a 400 Bad Request error. We're working on the issue, but meanwhile as a workaround, disabling caching via the LiteSpeed Cache plugin settings will bring sites back online.
Update 27/05/2020 08:45: Thanks for your patience on this, we've now put a fix in place and this should be resolved. Caching can safely be re-enabled where the above workaround was put in place temporarily.