| Server Offline (utley) |
|---|
| This server is currently offline and is under investigation. Further information will follow as it becomes available. Thank you for your patience in this matter. |
| Updated by Carl G-M. on 10th Nov 2010 @ 12:53pm |
| Server Offline (utley) |
|---|
| Just to confirm the RAID array degraded and the server has gone offline when the drive was swapped which we are investigated now. |
| Updated by Carl G-M. on 10th Nov 2010 @ 12:53pm |
| Server Offline (utley) |
|---|
| Server is coming back up, we are investigating why replacing the drive caused this and will be trying a different drive shortly to start rebuilding the RAID array. |
| Updated by Carl G-M. on 10th Nov 2010 @ 13:02pm |
| Server Offline (utley) |
|---|
| A different hard drive has been used and it's rebuilding but we are monitoring this closely. |
| Updated by Carl G-M. on 10th Nov 2010 @ 13:12pm |
| Server Offline (utley) |
|---|
| Server has gone into read only mode, we are investigating this further. |
| Updated by Carl G-M. on 10th Nov 2010 @ 13:14pm |
| Server Offline (utley) |
|---|
| Server is coming back up now. We are going to investigate the RAID issues further. |
| Updated by Carl G-M. on 10th Nov 2010 @ 13:24pm |
| Server Offline (utley) |
|---|
| Boot up has failed, we are working on this and hope to have further information shortly. |
| Updated by Carl G-M. on 10th Nov 2010 @ 13:27pm |
| Server Offline (utley) |
|---|
| Just an update and sorry for the delay! We've moved the working drive into another blade and it's responding again. We are auditing further before trying to rebuild RAID array. |
| Updated by Carl G-M. on 10th Nov 2010 @ 13:59pm |
| Server Offline (utley) |
|---|
| Once again sorry for the continued issues. I'm sorry to say the server has gone down again during rebuild (in different blade with different raid controller). We are getting the new drive removed and hopefully the server booted back shortly. |
| Updated by Carl G-M. on 10th Nov 2010 @ 14:12pm |
| Server Offline (utley) |
|---|
| We are shutting down SMTP, so email received will be queued on the cluster for the time being until it's restarted up. |
| Updated by Carl G-M. on 10th Nov 2010 @ 14:26pm |
| Server Offline (utley) |
|---|
| I can now confirm that this issue is resolved. In a very rare set of circumstances it appears that both drives in the RAID mirror failed to varying degrees so that we couldn't just replace a single failed drive for the array to rebuild. We managed to get the server up using just one of the failed drives (the least severe failure) and while in this state took a live image backup of the server during which time websites remained online and new emails left to queue on our email cluster. After the backup was taken this was restored to two brand new drives in a different chassis again, while leaving websites online from the broken server. This worked well and once the restore was complete the chassis containing the broken drive was powered off with the new one powered up which brought everything back to stability with data that was only minutes old. By the time you read this all queued emails will have been delivered. Apologies to all clients affected by this. |
| Updated by Chris James on 10th Nov 2010 @ 15:40pm |