Server Offline (pierce)
This server is currently offline and is under investigation. Further information will follow as it becomes available.
Updated by Carl G-M. on 5th Jul 2018 @ 12:39pm

Server Offline (pierce)
Server is still completely unresponsive, we are investigating a hardware fault and hope to have an update shortly.
Updated by Carl G-M. on 5th Jul 2018 @ 12:50pm

Server Offline (pierce)
Server is back up, we are monitoring this closely and further action maybe required. We will update this announcement with any further information.
Updated by Carl G-M. on 5th Jul 2018 @ 12:58pm

Server Offline (pierce)
We are continuing to experience issues with this server. We will update as soon as we have further information.
Updated by Simon Pendlebury on 5th Jul 2018 @ 14:55pm

Server Offline (pierce)
Unfortunately after investigating this further we have determined that this server needs to be migrated to new hardware before service can be restored - the team are in the process of deploying and configuring the new hardware, which we expect to be finished shortly, at which point we will begin the process of restoring account data to the new server.
Updated by Sam Pizzey on 5th Jul 2018 @ 19:07pm

Server Offline (pierce)
Migration of this server to the new hardware is still in progress. Due to the amount of data which needs to be transferred, this is unlikely to be complete before the morning.
Updated by Ben H. on 6th Jul 2018 @ 00:17am

Server Offline (pierce)
Still no significant information to pass on yet. Just confirming that work on restoring this server is still in progress.
Updated by Ben H. on 6th Jul 2018 @ 06:00am

Server Offline (pierce)
Due to various hardware issues this has taken longer than anticipated. We hope to have the server up and running within the hour, and restoration of account data will begin immediately after that.
Updated by Simon Pendlebury on 6th Jul 2018 @ 09:32am

Server Offline (pierce)
The server is up and running. We are now restoring account data.
Updated by Simon Pendlebury on 6th Jul 2018 @ 11:16am

Server Offline (pierce)
A number of accounts are now restored and functional again. We're still working on the remainder.
Updated by Simon Pendlebury on 6th Jul 2018 @ 14:34pm

Server Offline (pierce)
All accounts have now been restored and services are fully operational. There may be a slight delay until emails delivered during the outage arrive, as the queue backlog is processed. Once again, apologies for any inconvenience caused during this outage If you experience any further issues at all, please open a ticket and we will investigate immediately.
Updated by Sam Pizzey on 6th Jul 2018 @ 20:58pm

Server Offline (pierce)
RFO (Reason For Outage) Report Date: Thursday 5th July, 2018 Server: pierce.uk-noc.com Problem description: Server hardware failure Impact: Sites on server unavailable. At approximately 12:30pm, server instability was detected through internal monitoring systems. The server became unresonsive soon after this, was rebooted and became available again. At 2:30pm, we detected further issues, with some websites being unavailable. Due to repeated loss of console access to the server, it was initially believed that there may have been malicious activity, and network connectivity was disabled. However, upon further investigation it was discovered that the issue was related to a critical hardware failure, and no account data had been compromised. The backup system was mounted on a spare server and determined to be fully intact. Work began on provisioning a replacement server, though due to various hardware issues, this took substantially longer than expected. At approximately 11am on 6th July, the restoration of account data began. Websites were made available over the course of the day, with the restore process completing at 9pm. All email for recipients on the server was queued on the email cluster, so in order to prevent these from bouncing, mail services were disabled until the account restore was complete. We appreciate the disruption caused by this incident, and in light of this, measures have been taken to decrease the time required to provision replacement hardware in the unlikely case that similar issues recur. We will be crediting all accounts hosted on this server with a month's free hosting, without prejudice.
Updated by Simon Pendlebury on 11th Jul 2018 @ 10:06am