“I am desperately sorry for the recent proofing service downtime”.
A short sentence I know but a statement from me that is heartfelt and genuine and is not meant to be in any way flippant, it does however say exactly what needs to be said.
The last couple of weeks have been a testing time for all of us here at Everybodysmile and not a period that I’m in any hurry to repeat!
Our service level is always something that I have held to be our number one priority and the recent downtime is something that I am extremely embarrassed about. I completely understand the impact it has on your business and please believe me when I say that I am deeply sorry for the problems it causes you.
I have been running the company for ten years now and I can honestly say that during this time, we have not suffered anything like this type of ‘on/off’ service and we are doing everything we possibly can to put things in place to ensure that things run smoothly from now on.
That said, we are not entirely through the choppy waters and I ask you to please be patient and bear with us while we adapt our systems and work towards an improved solution.
We employ a superb team to maintain our servers but despite their best efforts, they have been powerless to stop unexpected write failures on two of our disks. The failure to write successfully to these disks has caused our main server to switch to a ‘read only’ state and this then has to be re-mounted correctly which takes time and has been the cause of the recent downtime.
At this moment, it hasn’t been possible to complete the work needed to entirely replace the disks and this work is still being undertaken. This will inevitably lead to further downtime but we will attempt to keep this to an absolute minimum and carry out the work overnight.
We have always worked to a plan for hardware replacement but this incident has forced us to bring forward work that had been sheduled for a few months time.
Along with replacing the affected disks, we have been working towards an improved solution based on the Amazon Web Services platform and will be migrating all of our image data in the coming weeks. This will provide us with a more resilient data store and will reduce the dependence on physical hard drives in our current servers.
We will also be looking to introduce a server failover system to help alleviate downtime even further but all of this does of course take time to implement.
Once again I offer my sincere apologies for the intermittent service and please rest assured that we are doing all we can to continue to improve our service in all areas.