| Server down |
|---|
| This server (Ace) developed a problem a short time ago and despite best efforts to stabalise, has needed to be power cycled. It is currently booting back up which is delayed while a filesystem check takes place. All services should be back up very soon and this announcement will be updated when this happens. |
| Updated by Chris James on 12th Jul 2009 @ 02:02am |
| Server down |
|---|
| Some services are coming back up but there is still a problem on the server. We are continuing to work on it. |
| Updated by Chris James on 12th Jul 2009 @ 02:17am |
| Server down |
|---|
| Unfortunately an amount of data has been corrupted on one of the server partitions. It will be taken down in a few minutes to do a bare metal restore of this partition from the last backup (which is only an hour old). |
| Updated by Chris James on 12th Jul 2009 @ 02:22am |
| Server down |
|---|
| The /var partition is at 25% complete on the restore. |
| Updated by Chris James on 12th Jul 2009 @ 02:43am |
| Server down |
|---|
| The restore of /var is complete and the server is booting up where we'll see if it worked successfully. Another filesystem check will also be happening. |
| Updated by Chris James on 12th Jul 2009 @ 03:25am |
| Server down |
|---|
| Unfortunately a filesystem check of /home is required which contains a lot of data. This will take some time. |
| Updated by Chris James on 12th Jul 2009 @ 03:35am |
| Server down |
|---|
| The filesystem check on /home is now complete and the bootup is finished. The restored /var partition now looks fine so we're plugging it back into the network now after which everything should be back online. |
| Updated by Chris James on 12th Jul 2009 @ 03:53am |
| Server down |
|---|
| The server was plugged back in a short time ago and all services are up. Apologies for the downtime during this issue. In case it is any consolation, this kind of problem would have involved a 4-6 hour complete server restore in the past. It is thanks to our fairly new level 2 backup systems that meant we could do a bare metal restore of the partition and greatly improve the speed and ease of recovery. |
| Updated by Chris James on 12th Jul 2009 @ 04:08am |