Clook Status - Urgent Maintenance Notice

RESOLVED
This announcement has been resolved, no further updates are expected.

Urgent Maintenance Notice
This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime. We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem. As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems. Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following: 1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance. 2. At the time of the work doing a final sync to copy any new/changed data. 3. Turning off the old server and bringing up the same IP addresses on the new hardware. 4. Letting the new server take over serving websites for hosted accounts. This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace. Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware has been provisioned today and the first data sync is currently taking place. The final sync will be completed tomorrow, with the actual switch of IP addresses being completed tomorrow between 12pm and 2pm. This announcement will be updated before and after the work. Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.
Updated by Anthony German on 30th Nov 2021 @ 14:14pm

Urgent Maintenance Notice

This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime.

We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem.

As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems.

Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following:

1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance.
2. At the time of the work doing a final sync to copy any new/changed data.
3. Turning off the old server and bringing up the same IP addresses on the new hardware.
4. Letting the new server take over serving websites for hosted accounts.

This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace.

Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware has been provisioned today and the first data sync is currently taking place. The final sync will be completed tomorrow, with the actual switch of IP addresses being completed tomorrow between 12pm and 2pm. This announcement will be updated before and after the work.

Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.

Updated by Anthony German on 30th Nov 2021 @ 14:14pm

Urgent Maintenance Notice
The work is starting on this now
Updated by Carl G-M. on 1st Dec 2021 @ 11:52am

Urgent Maintenance Notice
Server IP switchover is in progress.
Updated by Carl G-M. on 1st Dec 2021 @ 12:19pm

Urgent Maintenance Notice
Server is now being shutdown for hardware move.
Updated by Carl G-M. on 1st Dec 2021 @ 12:27pm

Urgent Maintenance Notice
Server is back up and we are now finishing up the misc. tasks.
Updated by Carl G-M. on 1st Dec 2021 @ 12:34pm

Urgent Maintenance Notice
This is now all done. Please open a support ticket with any outstanding issues.
Updated by Carl G-M. on 1st Dec 2021 @ 12:43pm

03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.

03300 885 250