| Server issues |
|---|
| This is a follow on from the high load / server issues announcement earlier today. We are working on resolving this issue as quickly as possible, the first step we are taking to ensure data integrity is restoring our last backup taken to new hardware and then looking at resyncing any more recent data from the current server. |
| Updated by Carl G-M. on 27th Feb 2021 @ 10:46am |
| Server issues |
|---|
| Just a quick update, to confirm we have started with the last backup restore to new hardware. Once this is done, we will be switching the existing IPs over and then doing some final steps after that to resume service on the new server. |
| Updated by Carl G-M. on 27th Feb 2021 @ 11:09am |
| Server issues |
|---|
| Our System Admins are still working on restoring the data to the new hardware and more updates will be provided as they come available. |
| Updated by Alex P on 27th Feb 2021 @ 23:21pm |
| Server issues |
|---|
| The re-sync of data from the original server is now underway. To protect the integrity of the data and ensure nothing is missed, we are going to disable access to websites. Your sites will be unavailable until the new server is live. As always, updates will follow as they become available. Thank you for your patience with this matter. |
| Updated by James Scott on 28th Feb 2021 @ 11:20am |
| Server issues |
|---|
| The re-sync is still ongoing, sites beginning in a-l are now complete. |
| Updated by James Scott on 28th Feb 2021 @ 13:24pm |
| Server issues |
|---|
| The re-sync is now complete and we are working on setting up the networking on the new box. |
| Updated by James Scott on 28th Feb 2021 @ 15:24pm |
| Server issues |
|---|
| The new server is now live and the issues which led to this downtime have been resolved with the migration. We're working on clean-up now and will update this ticket once done. |
| Updated by James Scott on 28th Feb 2021 @ 16:44pm |
| Server issues |
|---|
| Our team completed the work on this server earlier today, the sites were online as of our announcement on: 2021-02-28 16:44:19 A full RFO will be provided as soon as we have all the details. |
| Updated by James Scott on 1st Mar 2021 @ 15:07pm |
| Server issues |
|---|
| RFO: Hammett.UK-NOC.com - 02/03/2021 | Outage announcement: https://status.clook.net/view/4198-server-issues-7 | Title: Sever Issues | Date: 27/02/2021 - 28/02/2021 | Initial Investigation: Initial investigation of the servers performance issues (https://status.clook.net/view/4197-high-load-hammett-5) indicated the server suffered a RAID controller hardware failure. SysAdmins and datatcentre engineers began the work to remove the drives from the old server chassis to a new chassis. SysAdmins continued to monitor the issue which was seemingly resolved until 27/02/2021 when performance issues reappeared. | Root Cause: Fatal disk failure in the server caused disk responsiveness to slow and become service-affecting. | Actions Taken: Data was transferred to new disks in a new chassis to rule out any further hardware failures. Data was taken from the newest backup available (26/02/2021) with a re-sync from the live server disks. There were no issues with transferring the data to the new chasssis. The SysAdmin team continued to clean up the new server to match the old server, this was completed with no issues. | Further Work: No further work is required on this server/hardware. |
| Updated by James Scott on 2nd Mar 2021 @ 14:06pm |