Server

Onyx Network Status

Clear communication throughout service affecting events

View latest announcements
Some systems are experiencing issues

Past Incidents

29th May 2020

RFO for recent performance issues

Impact: We have been investigating an issue where sites are intermittently seeing 400 errors when trying to load assets within a site such as images, CSS or JavaScript. This was found to be caused by a faulty update within the third-party software, LiteSpeed ADC, powering our caching layer. Additionally, we encountered an unrelated issue with a small number of sites on our database cluster using a larger than normal number of long-running database queries, in turn leading to database performance issues more widely across Onyx.

Remedial Steps: For the LiteSpeed issues, we rolled back to an earlier version to resolve the issues whilst we work with LiteSpeed to resolve this issue in the latest version of the software. The database issues have been tracked down to a faulty plugin on the sites concerned, and we have worked with the clients to remove this and find alternatives whilst the plugin vendor patches the issue. Additionally, we have taken the opportunity to boost database capacity by bringing online additional clusters, and put the sites with the busiest queries on their own cluster to minimise the risk of this recurring. Our team are also developing additional layers of caching to enhance object and query performance.

28th May 2020

No incidents reported

27th May 2020

Database performance issues

We're aware of an issue with reduced database performance and are working on this now. We'll provide an update shortly.

Edit: 18:49 - Performance has returned to normal levels and we're continuing to monitor. Thanks for your patience.

Slow database queries

We're aware of there being an issue with slow queries on the database cluster, we hope to have this resolved shortly.

Edfit 15:09: This is now resolved and we're continuing to monitor. We're in the process of implementing a long-term fix for this now and apologise for any inconvenience caused.

26th May 2020

Caching is causing sites to not load correctly

We're aware of an issue where our caching layer is causing some elements of a site to not load, showing a 400 Bad Request error. We're working on the issue, but meanwhile as a workaround, disabling caching via the LiteSpeed Cache plugin settings will bring sites back online.

Update 27/05/2020 08:45: Thanks for your patience on this, we've now put a fix in place and this should be resolved. Caching can safely be re-enabled where the above workaround was put in place temporarily.

25th May 2020

No incidents reported

24th May 2020

No incidents reported

23rd May 2020

No incidents reported