| This server (Ace) developed a problem a short time ago and despite best efforts to stabalise, has needed to be power cycled. It is currently booting back up which is delayed while a filesystem check takes place.
All services should be back up very soon and this announcement will be updated when this happens.
|Updated by Chris J. on 12th Jul 2009 @ 02:02am|
|Some services are coming back up but there is still a problem on the server. We are continuing to work on it.|
|Updated by Chris J. on 12th Jul 2009 @ 02:17am|
|Unfortunately an amount of data has been corrupted on one of the server partitions. It will be taken down in a few minutes to do a bare metal restore of this partition from the last backup (which is only an hour old).|
|Updated by Chris J. on 12th Jul 2009 @ 02:22am|
|The /var partition is at 25% complete on the restore.|
|Updated by Chris J. on 12th Jul 2009 @ 02:43am|
|The restore of /var is complete and the server is booting up where we'll see if it worked successfully. Another filesystem check will also be happening.|
|Updated by Chris J. on 12th Jul 2009 @ 03:25am|
|Unfortunately a filesystem check of /home is required which contains a lot of data. This will take some time.|
|Updated by Chris J. on 12th Jul 2009 @ 03:35am|
|The filesystem check on /home is now complete and the bootup is finished. The restored /var partition now looks fine so we're plugging it back into the network now after which everything should be back online.|
|Updated by Chris J. on 12th Jul 2009 @ 03:53am|
| The server was plugged back in a short time ago and all services are up.
Apologies for the downtime during this issue. In case it is any consolation, this kind of problem would have involved a 4-6 hour complete server restore in the past. It is thanks to our fairly new level 2 backup systems that meant we could do a bare metal restore of the partition and greatly improve the speed and ease of recovery.
|Updated by Chris J. on 12th Jul 2009 @ 04:08am|