03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
RESOLVED
This announcement has been resolved, no further updates are expected.
Rebooting Server (richards)
This server is going down for a reboot and will be back up shortly. Thank you for your patience in this matter.
Updated by Eric B. on 24th Oct 2011 @ 04:35am
Rebooting Server (richards)
After rebooting this server has not come back online. We are currently investigating the matter further and provide more details shortly.
Updated by Eric B. on 24th Oct 2011 @ 05:25am
Rebooting Server (richards)
We've isolated the problem as an issue with the current kernel on the server. We are actively investigating a solution and will continue to provide updates as they are available.
Updated by Eric B. on 24th Oct 2011 @ 06:38am
Rebooting Server (richards)
Unfortunately the operating system on this server is beyond repair and we need to do a restore from backup. This work is being prepared now and will consist of an operating system and control panel reinstall then user data restored from backup.

This will unfortunately take some hours and this announcement will stay updated.
Updated by Chris J. on 24th Oct 2011 @ 08:26am
Rebooting Server (richards)
The OS and cPanel install has started. User data is going to get restored from backup which is started at the same time.
Updated by Chris J. on 24th Oct 2011 @ 09:32am
Rebooting Server (richards)
The OS and cPanel install are complete. Unfortunately the situation has gone from bad to worse as the backup drive is not reading data which we are working on now. At this time it is unclear whether the backup data will be recoverable
Updated by Chris J. on 24th Oct 2011 @ 10:21am
Rebooting Server (richards)
Better news - we have been able to recover the data on the backup drive. User data restoration has started
Updated by Chris J. on 24th Oct 2011 @ 10:38am
Rebooting Server (richards)
Backup data is now being restored. As user home directories get restored sites should start coming back online meaning that users starting with the letter A will come back online before those starting with Z. The file restore will take a while to complete due to the amount of data but it is underway.

We have blocked new emails coming in until all restores are complete. Following this, email will be restarted and all received emails that have been queued on our cluster will come through.
Updated by Chris J. on 24th Oct 2011 @ 10:58am
Rebooting Server (richards)
User data is approx 25% restored and the process is continuing with sites coming back online as that username gets restored.
Updated by Chris J. on 24th Oct 2011 @ 11:55am
Rebooting Server (richards)
User home directory restores are coming up to 50% complete. As a correction to our last update, websites are not yet back online as we're recompiling Apache and repairing some MySQL databases. Once these two tasks are complete all websites for users already restored will start to work.

Unfortunately it's not possible to give an accurate estimate of time of completion due to the number of factors involved but all user data should be restored within the next 60-90 minutes though this is subject to change.
Updated by Chris J. on 24th Oct 2011 @ 12:21pm
Rebooting Server (richards)
User directories are still restoring at a good rate but we're having problems with the Apache compile which we're continuing to work on.
Updated by Chris J. on 24th Oct 2011 @ 13:25pm
Rebooting Server (richards)
We have unfortunately hit further problems with libraries on the server and the situation has worsened. It is unclear at this stage whether the problem can be repaired and restores can continue or whether we will need to start over. Sorry for the additional bad news.
Updated by Chris J. on 24th Oct 2011 @ 14:23pm
Rebooting Server (richards)
We're still trying to get the restored server booted using boot discs so a repair on the libraries can be worked on. It is still unclear whether the restore from backup will need to be started again.
Updated by Chris J. on 24th Oct 2011 @ 15:16pm
Rebooting Server (richards)
I'm sorry to bring more bad news but we're in the worst case scenario and we need to start the restore again. This will unfortunately take another 3-5 hours.

Emails are still being queued so anything received will not be lost.
Updated by Chris J. on 24th Oct 2011 @ 16:21pm
Rebooting Server (richards)
The os is now installed and we're starting the cPanel install now at the same time as starting user home directories.
Updated by Chris J. on 24th Oct 2011 @ 16:58pm
Rebooting Server (richards)
User home directories are approx 40% restored. The cPanel install is still happening.
Updated by Chris J. on 24th Oct 2011 @ 18:03pm
Rebooting Server (richards)
User home directories are approx 75% complete.
Updated by Chris J. on 24th Oct 2011 @ 19:10pm
Rebooting Server (richards)
All user data is now restored and we're just doing a final Apache compile before services will be brought up. Following this all services will be online and we'll wrap up the work in the background before doing a final server reboot to complete the restore.
Updated by Chris J. on 24th Oct 2011 @ 20:24pm
Rebooting Server (richards)
All services are now back online and the email queue is coming through. We are continuing to work on some other items while services remain up.
Updated by Chris J. on 24th Oct 2011 @ 21:21pm