03300 885 250

Technical Infrastructure Status

We believe in full transparency, everything you see here is 100% live.
RESOLVED
This announcement has been resolved, no further updates are expected.
Server Down
We are aware of an issue at this time with the server Martyn and are currently working to resolve the situation. Thank you for your patience in this matter.
Updated by Eric B. on 4th Sep 2009 @ 17:23pm
Server Down
This has occurred again and we are investigating this. Sorry for the continued issues.
Updated by Carl G-M. on 5th Sep 2009 @ 11:44am
Server Down
Server is being restarted to try and resolve this network issue and we hope to have more information shortly.
Updated by Carl G-M. on 5th Sep 2009 @ 11:58am
Server Down
Server is coming back up and the attack is continuing, so we are filtering httpd activity right now but e-mail should be working fine.
Updated by Carl G-M. on 5th Sep 2009 @ 12:04pm
Server Down
I'm trying something new, which should hopefully take effect over the new 20-30 minutes as the DNS updates.
Updated by Carl G-M. on 5th Sep 2009 @ 13:51pm
Server Down
Sorry for the lack of updates, all the DNS changes are going through still but I hope to confirm this is all complete over the next 10 minutes and hopefully the attack will not continue to follow.
Updated by Carl G-M. on 5th Sep 2009 @ 15:10pm
Server Down
I can confirm the DNS changes have gone through and the sites should be coming back online now depending on your ISP/Router DNS.
Updated by Carl G-M. on 5th Sep 2009 @ 15:12pm
Server Down
At ~11am GMT the monitoring server picked up an issue with Martyn and I investigated this and found the cause to big a DDos attack aimed at the port 80 only and tried to filter this out but due to size this was not possible at either server level or network level. After rulling out this as a viable option I then went a tested to see if changing the DNS of any domains to a different IP would be viable for this attack and it appears to be the next option viable to try and this was done. However, during this period the attack continued and additional DNS technical issues meant that this process took a lot longer than it should do. We are continuing to monitor the server closely and hopefully this should be resolved.

I must say I am very sorry for the amount of time it took to completely return httpd availability but I wish to explain I was trying at all times to not cause another server crash and e-mail downtime and possible dataloss. That said, I have found a few key points where things could have been improved and also during this timeperiod hopefully managed to streamline this process down to <20-30 minutes including DNS changes should this situation occur again.
Updated by Carl G-M. on 5th Sep 2009 @ 15:20pm
Server Down
Unfortunately we are experiencing issues similar to yesterday but we are working to resolve this situation permanently.
Updated by Clook Internet on 6th Sep 2009 @ 13:39pm
Server Down
The DNS issues we were experiencing today have been sorted, and all but a handful of sites should be back online and resolving properly.
Updated by Chris B. on 7th Sep 2009 @ 04:30am
Server Down
It appears we are still experiencing problems with Dedicated IPs (IE sites with SSL certificates) and Also with some Addon domains.

We apologise profusely about these on going problems and are doing everything we can do ensure this server is back running smoothly ASAP
Updated by Clook Internet on 7th Sep 2009 @ 08:42am
Server Down
Please note due to the way this attack has been filtered it has come to our attention that some sites will be pointing incorrectly to other sites on the server.

This is a result of some IP/DNS changes and rest assured we are aware of and looking into these issues.
Updated by Clook Internet on 7th Sep 2009 @ 09:52am
Server Down
All customers using external DNS providers (IE their domain is not using the nameservers ns1.clook.net and ns2.clook.net) will need to contact us ASAP as their external DNS entries will need to be edited.

We are continuing to work on the remaining problems.
Updated by Clook Internet on 7th Sep 2009 @ 11:29am
Server Down
We have made progress with this problem and you may now see that your website is back online.

However there are still one or two issues that need to be ironed out and we'll continue to keep you updated via this announcement.

Thank you for your continued patients.
Updated by Clook Internet on 7th Sep 2009 @ 14:40pm
Server Down
All issues should now be resolved and we are waiting for DNS propagation due to the IP changes that have been made.

This should take 10 / 15 minutes but may take a little longer depending on your ISP and also local DNS cache.
Updated by Clook Internet on 7th Sep 2009 @ 15:06pm
Server Down
This has started again, we are trying to track this down now. I'll update asap.
Updated by Carl G-M. on 8th Sep 2009 @ 12:02pm
Server Down
This should now be resolved, although we are continuing to monitor this closely.
Updated by Carl G-M. on 8th Sep 2009 @ 12:22pm
Server Down
As an update to this situation, mid yesterday we took the decision to put each website on this server on it's own IP address so that should the attack happen again we can determine the site being targetted (often impossible when multiple sites share IP space).

Earlier today a new attack came in and thanks to the above we were able to pin it to a specific domain name which after being taken offline (ip and single site) stopped the attack. All has been very stable since this time and we are hopeful of a long term resolution.

We will need to change IP space back but will be leaving this for 1-2 weeks and will of course schedule maintenance to do this at an off peak time and with measures in place to handle it seamlessly.

We apologise for the inconvenience suffered to all clients during the last few days while this has been going on. The matter is far from closed with us and there will be meetings with datacentre staff to discuss possible methods of preventing or easing a situation like this with different network infrastructure in the future.

By way of apology we will be adding 2 months to the term of all hosting packages on this server. This will be done during this week before the next invoice run.
Updated by Chris James on 8th Sep 2009 @ 15:36pm