Over the past 24 hours there have been precisely 3 responses from ServerAxis whom we trusted to provide service and networking within Miami-IX (datacentre). Initially they were looking in to it as at this point it was incorrectly assumed to be power related due to the nature of the failure. Despite numerous attempts to gain updates very little was forthcoming until confirmation was sent that it was a networking issue affecting all servers and services.
Many further attempts were made to glean more information including around 100 phone calls, none of which were answered. A final update came from ServerAxis around 6 hours ago stating they are still checking.
Obviously, this level of service is completely unacceptable. It was then discovered that the entire /20 subnet was dropped by Cogent (network upstream) and it is assumed that there is a financial dispute between Cogent or Miami Internet Exchange directly and ServerAxis waiting to be resolved.
With this in mind and no predictable ETA in sight for getting things restored, I have been liaising with other hosts to get alternative hardware provisioned within Florida. The best time scale offered in Florida due to the hardware spec. required is 14 days which is obviously not acceptable. Therefore requests were sent further afield. The chosen solution is to deploy a new server in Phoenix, Arizona and recreate all user accounts there.
Getting the new hardware, reissue all licenses and installing and configuration of the server will take some time, but please be assured that everything that can be done is being done.
ServerAxis were initially selected due to their long record in business and has serverd us well during the three years shared hosting has been offered. This has come as a complete surprise and one that could not be predicted. It has left us, our customers and our customer’s customers in a bad and in some cases vulnerable position, with a completely unacceptable response.
Due to data volumes and frequency it was impractical to have offsite backups and the local backup server is also a victim to the same issue.
If data can be recovered when server access is restored, it will be. Plan for the worst and hope for the best.
A new welcome email will be sent out as soon as your service is ready. Unfortunately this service will be completly empty and you will need to start fresh again.
If you instead would like to be deployed on the Amsterdam, Netherlands server please open a ticket with 'request to move my service to Netherlands' in the subject line.
This offer is NOT available to you who uses the free sub-domain option.
I would like to apologise again for the inconvenience this will no doubt cause to many people but due to the nature of the problem it is impossible to mitigate the impact in any other way.
Further instructions will follow where necessary.
Wednesday, March 2, 2016
