Clook Status - Urgent Maintenance Notice

RESOLVED
This announcement has been resolved, no further updates are expected.

Urgent Maintenance Notice
This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime. We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem. As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems. Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following: 1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance. 2. At the time of the work doing a final sync to copy any new/changed data. 3. Turning off the old server and bringing up the same IP addresses on the new hardware. 4. Letting the new server take over serving websites for hosted accounts. This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace. Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware has been provisioned and the first data sync will be completed tomorrow. A final sync of the data, will also be completed tomorrow, with the actual switch of IP addresses being completed tomorrow between 12pm and 3pm. This announcement will be updated before and after the work. Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.
Updated by Anthony German on 2nd Dec 2021 @ 16:21pm

Urgent Maintenance Notice

This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime.

We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem.

As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems.

Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following:

1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance.
2. At the time of the work doing a final sync to copy any new/changed data.
3. Turning off the old server and bringing up the same IP addresses on the new hardware.
4. Letting the new server take over serving websites for hosted accounts.

This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace.

Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware has been provisioned and the first data sync will be completed tomorrow. A final sync of the data, will also be completed tomorrow, with the actual switch of IP addresses being completed tomorrow between 12pm and 3pm. This announcement will be updated before and after the work.

Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.

Updated by Anthony German on 2nd Dec 2021 @ 16:21pm

Urgent Maintenance Notice
Just an update, the IP switchover has been delayed but we are progress still on the data sync and will update with an ETA shortly.
Updated by Carl G-M. on 3rd Dec 2021 @ 14:18pm

Urgent Maintenance Notice
Final sync in progress.
Updated by Carl G-M. on 3rd Dec 2021 @ 14:31pm

Urgent Maintenance Notice
Final sync is done and IP switchover is in progress.
Updated by Carl G-M. on 3rd Dec 2021 @ 14:33pm

Urgent Maintenance Notice
Server is being shutdown for server hardware move.
Updated by Carl G-M. on 3rd Dec 2021 @ 14:41pm

Urgent Maintenance Notice
Server is back up, we are now doing misc. tasks.
Updated by Carl G-M. on 3rd Dec 2021 @ 14:48pm

Urgent Maintenance Notice
All work is now complete, please open a support ticket with any issues.
Updated by Carl G-M. on 3rd Dec 2021 @ 14:55pm

03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.

03300 885 250