03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
RESOLVED
This announcement has been resolved, no further updates are expected.
AT-RISK: Emergency Power Maintenance
This is notification of emergency maintenance that will occur today.

We will be replacing a Power Distribution Unit in the rack your server is hosted in. This means that the redundancy of the power in this rack will be lost during the confirmed time slot and the status of this announcement is detailed as AT-RISK.

Due to the nature of the works, qualified personnel will be present in the server rack at the time to handle the replacement.

---
Start: Thursday 13th July @ 12:00 PM BST
End: Thursday 13th July @ 12:30 PM BST
---

The above is not a notice of downtime, however is an estimated time of when the works will start and end.
We expect the works to take upwards of 15 to 20 minutes, however we have allocated a larger window to accommodate any potential issues.

We will update this announcement once the works are due to commence and again as they are subsequently completed.
Updated by Dan Miller on 13th Jul 2017 @ 11:35am
AT-RISK: Emergency Power Maintenance
These works are about to commence.
Updated by Dan Miller on 13th Jul 2017 @ 11:59am
AT-RISK: Emergency Power Maintenance
We have encountered an issue and engineers are reverting these works.

We anticipate to get servers online as soon as possible.
Updated by Dan Miller on 13th Jul 2017 @ 12:22pm
AT-RISK: Emergency Power Maintenance
All services should be back online and functional.

Despite the removal of redundancy, we are investigating the cause of the downtime and will provide a Reason for Outage (RFO) shortly.
Updated by Dan Miller on 13th Jul 2017 @ 12:34pm
AT-RISK: Emergency Power Maintenance
The work is now completed fully.

We have detailed the Reason for Outage below.

--
Dear Customers,

During a planned maintenance slot, we encountered full power loss to a rack which contains your service.

Upon removal of the first PDU from the rack, engineers were disconnecting this device from the floor pit when the rack lost power input. It was discovered that the redundant PDU which was serving power to the rack was labelled incorrectly, and subsequently was removed from the power distribution lane.

Engineers took this opportunity to swap out the Power Distribution Unit quickly, as the time taken to reconnect the existing PDU and to replace it was negligible.

To summarise, the downtime today was due to a labelling error, of which these devices were labelled incorrectly six years ago.

To mitigate this in future we have requested the on-site engineers to perform a full non-intrusive audit of power distribution lane labels to ensure these are marked correctly.

We apologise for any inconvenience caused during this momentary downtime.
--
Updated by Dan Miller on 13th Jul 2017 @ 13:32pm