03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
RESOLVED
This announcement has been resolved, no further updates are expected.
Scheduled reboot
This server (potter) unfortunately suffered a hard drive failure today with the RAID mirroring working as it should and not causing any downtime or service interruption. We have since replaced the failed drive with a new one but the data is not rebuilding to the new drive automatically as it should do.

In order to solve this we need to reboot the machine to force a RAID rebuild. Total downtime should not exceed 3-5 minutes and this is scheduled for 10pm this evening (Saturday 24th April).

This announcement will remain updated.
Updated by Chris James on 24th Apr 2010 @ 16:16pm
Scheduled reboot
This is a reminder that the reboot on this server will be happening in 15 minutes.
Updated by Chris James on 24th Apr 2010 @ 21:43pm
Scheduled reboot
The reboot is now complete and everything is back online.

Unfortunately the RAID array is still not rebuilding so we're continuing to investigate. Should any further work be required which impacts on services this announcement will be updated.
Updated by Chris James on 24th Apr 2010 @ 22:08pm
Scheduled reboot
In another attempt to fix this problem we need to schedule another quick reboot so that the blade can be powered down, reseated in the blade enclosure then powered back up. This should hopefully force the RAID rebuild.

This work is scheduled for 10:45pm (15 mins from now) and shouldn't involve more than 3-5 minutes downtime.

Thank you for your patience on this matter.
Updated by Chris James on 24th Apr 2010 @ 22:34pm
Scheduled reboot
The second reboot is now complete and the server is back online.

I'm happy to report that this fixed the problem and the RAID array has started rebuilding to the new drive. Thank you again to everyone on this server for your patience while this problem was resolved.
Updated by Chris James on 24th Apr 2010 @ 22:53pm