Clook Status - Urgent maintenance notice

RESOLVED
This announcement has been resolved, no further updates are expected.

Urgent maintenance notice
This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime. We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem. As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems. Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following: 1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance. 2. At the time of the work doing a final sync to copy any new/changed data. 3. Turning off the old server and bringing up the same IP addresses on the new hardware. 4. Letting the new server take over serving websites for hosted accounts. This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace. Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware is being provisioned today and the first data sync will take place tonight. The switch of IP addresses will be done tomorrow between 12pm and 2pm. This announcement will be updated before and after the work. Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.
Updated by Chris James on 22nd Nov 2021 @ 14:09pm

Urgent maintenance notice

This email is to notify of some upcoming maintenance on the server your account is hosted on which will involve a small amount of downtime.

We have been seeing some minor, largely unnoticeable filesystem corruption on some of our servers. Initially this appeared to be caused by a RAID card fault so we have been replacing this component to remedy the problem.

As the problem has continued, including on servers already with replaced RAID cards, we have been troubleshooting further. With the involvement of the hard drive manufacturer it now appears that the hard drives are at fault and it is thought that we have purchased a "bad batch" of drives in recent server deployments all having the same manufacturing fault. On the advice of the manufacturer, we will be removing all of this model of hard drive from our servers as a matter of urgency to solve the filesystem corruption problems.

Replacing all hard drives in a server would usually involve significant downtime while the server is reinstalled and data restored from backup. Instead, we will be minimising downtime and inconvenience by doing the following:

1. Provisioning a new server of identical specification on the network and syncing all user data to it in advance.
2. At the time of the work doing a final sync to copy any new/changed data.
3. Turning off the old server and bringing up the same IP addresses on the new hardware.
4. Letting the new server take over serving websites for hosted accounts.

This work should largely be seamless and involve minimal downtime of approx 5 minutes while the IP addresses are switched over and another 5 minute period of downtime while the machine is physically put into place in the same rackspace.

Due to the urgent nature of this work we will be carrying it out as soon as possible. The new hardware is being provisioned today and the first data sync will take place tonight. The switch of IP addresses will be done tomorrow between 12pm and 2pm. This announcement will be updated before and after the work.

Apologies for the short notice period of this work especially as you have only recently been migrated to this machine. Unfortunately due to the urgent nature of removing the bad drives from our servers we need to proceed as soon as possible.

Updated by Chris James on 22nd Nov 2021 @ 14:09pm

Urgent maintenance notice
Just an update on this, the first sync has been completed and we are planning on starting the second sync at around 12 and do the switchover assuming no issues shortly after as scheduled.
Updated by Carl G-M. on 23rd Nov 2021 @ 10:29am

Urgent maintenance notice
The final sync will be starting shortly.
Updated by Carl G-M. on 23rd Nov 2021 @ 12:00pm

Urgent maintenance notice
Data sync is done and IP switchover is done. We are just validating this and then we will proceed with the hardware physical move.
Updated by Carl G-M. on 23rd Nov 2021 @ 12:30pm

Urgent maintenance notice
Server is shutdown and is being moved now and will be brought back up shortly.
Updated by Carl G-M. on 23rd Nov 2021 @ 12:38pm

Urgent maintenance notice
Server is back up and we are finishing up. Please open a support ticket with any issues on websites or email.
Updated by Carl G-M. on 23rd Nov 2021 @ 12:47pm

03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.

03300 885 250