03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
RESOLVED
This announcement has been resolved, no further updates are expected.
Server down
This server (Ace) developed a problem a short time ago and despite best efforts to stabalise, has needed to be power cycled. It is currently booting back up which is delayed while a filesystem check takes place.

All services should be back up very soon and this announcement will be updated when this happens.
Updated by Chris J. on 12th Jul 2009 @ 02:02am
Server down
Some services are coming back up but there is still a problem on the server. We are continuing to work on it.
Updated by Chris J. on 12th Jul 2009 @ 02:17am
Server down
Unfortunately an amount of data has been corrupted on one of the server partitions. It will be taken down in a few minutes to do a bare metal restore of this partition from the last backup (which is only an hour old).
Updated by Chris J. on 12th Jul 2009 @ 02:22am
Server down
The /var partition is at 25% complete on the restore.
Updated by Chris J. on 12th Jul 2009 @ 02:43am
Server down
The restore of /var is complete and the server is booting up where we'll see if it worked successfully. Another filesystem check will also be happening.
Updated by Chris J. on 12th Jul 2009 @ 03:25am
Server down
Unfortunately a filesystem check of /home is required which contains a lot of data. This will take some time.
Updated by Chris J. on 12th Jul 2009 @ 03:35am
Server down
The filesystem check on /home is now complete and the bootup is finished. The restored /var partition now looks fine so we're plugging it back into the network now after which everything should be back online.
Updated by Chris J. on 12th Jul 2009 @ 03:53am
Server down
The server was plugged back in a short time ago and all services are up.

Apologies for the downtime during this issue. In case it is any consolation, this kind of problem would have involved a 4-6 hour complete server restore in the past. It is thanks to our fairly new level 2 backup systems that meant we could do a bare metal restore of the partition and greatly improve the speed and ease of recovery.
Updated by Chris J. on 12th Jul 2009 @ 04:08am