03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
RESOLVED
This announcement has been resolved, no further updates are expected.
Scheduled maintenance
This announcement is applicable to all clients on our Cloud platform.

We have a maintenance window scheduled for this platform at 10pm on Monday 15th December 2014 which is scheduled to last for one hour.

During this maintenance we will be upgrading the underlying software that powers the Cloud platform and taking the opportunity to run some fixes for non service affecting bugs that have cropped up in recent weeks. As some of these fixes and upgrades are related to the storage systems this maintenance window will be service affecting for servers on the Cloud platform.

There will be up to 30 minutes of downtime to the cloud infrastructure (including servers hosted within it) during this maintenance window. We will aim to minimise hosted servers downtime but unfortunately it is unavoidable with the type of work that is required.

This announcement will be updated before, during and after the work.
Updated by Chris James on 12th Dec 2014 @ 04:30am
Scheduled maintenance
Unfortunately we need to postpone the scheduled work for 24 hours meaning that the work is now scheduled for 10pm on Tuesday 16th December 2014.

This announcement will be updated before the work starts.

Apologies for any inconvenience caused by the postponed timescale.
Updated by Chris James on 15th Dec 2014 @ 12:54pm
Scheduled maintenance
As a reminder, this work will be stating in around 30 minutes and lasting for one hour. There will be some downtime for active cloud servers during this window.
Updated by Chris James on 16th Dec 2014 @ 21:37pm
Scheduled maintenance
As an update to this, we unfortunately hit a few delays while shutting some servers down which needed fixing before continuing. With this complete, work has been continuing and we're now rebooting the hypervisor nodes. Once these come back up, cloud servers will be brought back up.
Updated by Chris James on 16th Dec 2014 @ 23:14pm
Scheduled maintenance
Some hypervisors have not booted back up which is required before the work can continue. The matter is being worked on as a matter of extreme urgency.
Updated by Chris James on 16th Dec 2014 @ 23:41pm
Scheduled maintenance
Cloud server boot ups have now started so all should be back online very soon.
Updated by Chris James on 17th Dec 2014 @ 00:01am
Scheduled maintenance
A number of cloud servers have gone into a filesystem check while booting due to the time since last reboot. This will unfortunately add time to the boot up time.
Updated by Chris James on 17th Dec 2014 @ 00:13am
Scheduled maintenance
We still have a hypervisor refusing to boot which is keeping the last few cloud servers offline. This is being troubleshooted at this time.
Updated by Chris James on 17th Dec 2014 @ 00:28am
Scheduled maintenance
We have lost one of the cloud hypervisors to a RAID issue so working through each cloud server still offline bringing it up from the mirrored drive on other hypervisors. This is fairly slow work but is working with those still offline hopefully back online within the next 30 minutes.
Updated by Chris James on 17th Dec 2014 @ 01:23am
Scheduled maintenance
All cloud servers were brought back online last night with work completed by the makers of the cloud software at around 4am.

We will be working closely with the vendor to establish why a failure of one of the hypervisors caused such downtime to some cloud servers requiring such a level of manual intervention to bring them back online.

Apologies to all affected by the extended maintenance window and the problems suffered within it.
Updated by Chris James on 17th Dec 2014 @ 09:56am