Clook Status - Urgent Maintenance Notice

RESOLVED
This announcement has been resolved, no further updates are expected.

Urgent Maintenance Notice
This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime. We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem. As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems. Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following: 1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance. 2. At the time of the work doing a final sync to copy any new/changed data. 3. Turning off the old server and bringing up the same IP addresses on the new hardware. 4. Letting the new server take over serving websites for hosted accounts. This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace. Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware has been provisioned today and the first data sync is currently taking place. Another sync will be completed tomorrow, with the actual switch of IP addresses being completed tomorrow between 12pm and 2pm. This announcement will be updated before and after the work. Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.
Updated by Anthony German on 25th Nov 2021 @ 20:59pm

Urgent Maintenance Notice

This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime.

We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem.

As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems.

Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following:

1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance.
2. At the time of the work doing a final sync to copy any new/changed data.
3. Turning off the old server and bringing up the same IP addresses on the new hardware.
4. Letting the new server take over serving websites for hosted accounts.

This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace.

Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware has been provisioned today and the first data sync is currently taking place. Another sync will be completed tomorrow, with the actual switch of IP addresses being completed tomorrow between 12pm and 2pm. This announcement will be updated before and after the work.

Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.

Updated by Anthony German on 25th Nov 2021 @ 20:59pm

Urgent Maintenance Notice
The final sync and switchover will be starting shortly.
Updated by Carl G-M. on 26th Nov 2021 @ 11:46am

Urgent Maintenance Notice
This is in progress.
Updated by Carl G-M. on 26th Nov 2021 @ 12:01pm

Urgent Maintenance Notice
The IP switchover has completed and we are finalising some misc. tasks and will be allowing new emails to come through shortly.
Updated by Carl G-M. on 26th Nov 2021 @ 12:25pm

Urgent Maintenance Notice
The server is being shutdown for the hardware move.
Updated by Carl G-M. on 26th Nov 2021 @ 12:30pm

Urgent Maintenance Notice
Server has been back online for over an hour now, please open a support ticket with any issues.
Updated by Carl G-M. on 26th Nov 2021 @ 13:56pm

03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.

03300 885 250