03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
IN PROGRESS
This announcement is in progress and relates to an ongoing issue.
Cloud platform disruption
We are aware of a major issue affecting the storage systems on the majority of our Cloud v2 servers. The storage cluster has been recovered and is now serving data again but a small number of Cloud servers are still not yet back online. Our monitoring is alerting us to this and we are working on affected servers at the moment.

An update will be sent when every server is back online. We will also be doing a full investigation of what happened throughout the night and will send full details when we have them.
Updated by Chris J. on 14th Nov 2020 @ 21:49pm
Cloud platform disruption
All servers affected by this issue should now be back online and functioning normally. We are continuing to monitor but feel free to contact us if you see any issues on your server.
Updated by Chris J. on 14th Nov 2020 @ 23:04pm
Cloud platform disruption
To update this announcement, since all servers were put back online during Saturday evening everything has been stable and we have been working at finding the cause of the problem. Work is still ongoing with this as it involves very large log files and conversations with third party software vendors but we hope to have something more detailed soon.

In summary, the storage cluster in the cluster lost connectivity with a large number of physical hard drives resulting in them being assumed as failed and removed from the cluster. Due to the amount of drives, this resulted in the cluster not having enough drives left available to rebalance data and continue serving disks. At the time of the issue we ruled out physical hardware problems with the drives and manually added them back to the storage cluster which resulted in all storage coming back online once this was complete and data had rebalanced.

We do not believe there is a high chance of this happening again in the immediate future and we're continuing to focus resources on finding the exact cause so that a fix can be implemented.
Updated by Chris J. on 17th Nov 2020 @ 08:40am