SAUK Outage Jan 2012
Welcome back, everyone 
As you should be aware by now, our hosting company, A Small Orange, suffered a major hardware failure across their cloud hosting platform last Friday. SAUK was unreachable as a direct result. We were told originally that they would be unable to recover any of our data. I then spent the weekend working on getting a brand new server up and running, with the intention of getting the site back online with the most recent backup that we had access to at that time (from the 14th January 2012). This came online briefly on Monday evening. Just after I got the site back online, ASO got in touch to let us know that they had been able to recover our data, and were in a position to recreate our old server environment on our new one. We have then had to wait for them to do just that, and that's why the board did not reappear until this morning.
Everything should be working as it was before the recent downtime, but there will likely be a few bits and pieces that will need tweaking. The board may have to close intermittently for a short while over the next few days if stuff needs doing. If you do happen to experience any problems using the board over the next few days or so, please post something here or send me a PM so I can take a look at your issue ASAP.
PS: Some info from ASO, for the techies...
Quote:
Hello,
As many of you are aware, the A Small Orange Cloud Hosting platform suffered a prolonged outage that began in the late afternoon (Eastern time) on Friday, January 20th. At the most basic level, this outage was due to a disk failure in the array.
The cloud storage configuration, which is made up of a 16 disk SAN array, can tolerate one drive failure and continue to operate normally. This automatic failover provides the reliability that has made cloud platforms popular. In last nightâ??s incident, however, the SAN array had two drive failures, which is beyond the tolerance fault level of the system.
Numerous attempts have been made to rebuild the array and regain access to the data, but so far those attempts have failed. Due to the fact that two drives were lost, we feel at this point that all data across the entire cloud platform is lost and is likely unrecoverable.
We are continuing to work on the storage system and with its manufacturers to attempt to recover the data, but we wanted to let you know what the situation is at this point.
We are deeply sorry for the downtime you have experienced so far. As a web hosting company, letting customers know that data has been lost and is unrecoverable is the worst thing we ever have to do. We stand behind the products we sell and, in the rare occasions when there are failures that have a potentially damaging affect on our customers, it is a difficult thing to have to share with you. We know you count on us, and we hate falling short of expectations. Your trust has helped us grow into the company we are today and we are deeply grateful.
We know you will have many questions as to why and how this happened, and we will be happy to answer whatever questions we can. Unfortunately, A Small Orange leased the cloud platformâ??s hardware from a third party datacenter for our customersâ?? use. In some cases, we may not be able to fully answer your questions until we are done investigating. The monitoring of the integrity of the hardware system and the disks were the responsibility of the data center we leased the hardware from. We are working to get answers regarding when the disk issues occurred, how they occurred, whether there was any notification of impending disaster and if that took place. While we can tell you what has happened to the system, unfortunately, we don\'t have answers yet as to why it happened. This is as frustrating for us as it likely is for you and we are sorry we do not have more information to share with you at this time.
Our lack of control over our current cloud hosting platform and our inability to monitor certain aspects of it (both due to the fact it was leased hardware), is what led us four months ago to invest significantly in the design of our own cloud platform. Many of you have gotten the emails regarding the migration that was due to begin this upcoming Monday. Our new cloud platform is wholly owned, controlled, and monitored by A Small Orange, and is designed to be fault tolerant to two disk drives vs. the current cloud set up, which is fault tolerant to only one failure. Our new cloud is currently up and working well and new customers are already being placed on it.
While we realize that there is little we can do to correct a situation as catastrophic as this, we do want to do whatever we can to help you recover from this situation. For those customers that would like to stay with A Small Orange, we will:
- provide you with 6 free months on the new cloud
- automatically upgrade you to the next highest package level at no cost
- offer you 2 GB of swap memory at no cost
- offer our R1Soft Bare Metal Backup Solution with multiple points of restore at no cost
You can find details on how to get your new cloud hosting account activated at: http://asmallorange.com/cloud-2012.php
We do realize that for some, this is a catastrophic failure that will result in your choosing to leave A Small Orange. While we hope that we can retain your trust and your business, we understand this won\'t be possible for everyone. If you choose to leave A Small Orange we will, of course, refund you any pre-paid fees as well as pay for your first six months of comparably priced hosting at a new hosting company. You can find details on this at: http://asmallorange.com/cloud-2012.php
We will be reaching out to many of you over the next several days by phone to discuss what has happened and to answer any questions you might have. Customer service continues to be our number one priority at A Small Orange and we are working hard to do whatever we can to make this situation right.
We again extend our deepest apologies.
Very truly yours,
The Team at A Small Orange
--
Douglas Hanna
A Small Orange LLC
|
Quote:
Early on Saturday, January 21st, we made the extremely difficult decision to let our customers know that the cloud platform we had been using had likely suffered a catastrophic and unrecoverable event that we believed had permanently destroyed our ability to bring up the system and subsequently read or access any customer data. Even though we had planned for a migration to our brand new cloud platform to start just two days later, we (and our customers) were out of luck. It was probably the worst news we could have shared with our customers that morning and it hurt.
While many of our employees, supervisors, and managers were completely focused on doing whatever we could to help our customers get over this catastrophic data loss, our top system administrators continued to work long hours with the manufacturers and engineers of the cloud platform searching for any potential solution. Any lead was worth pursuing and we were determined to do just that until we had exhausted every option available. It took well over a hundred man hours, but we are happy to report that a significant amount of data was able to be obtained from the old cloud and that we do indeed have a full backup of your previous virtual machine.
Again, to be clear - we have a full backup of your data as it existed at the time the cloud went down on Friday evening.
As many people made some changes to their cloud set up after we recreated accounts, we are reaching out individually to customers that already have new cloud accounts to ask them what they want us to do with the recovered data.
We can completely overwrite what you currently have and reproduce your data to its exact state previous to when the cloud went offline (option 1), or we can make this data available to you and you can make changes as you see fit (option 2). Completely overwriting your current virtual machine with the backup is the fastest and easiest way to get your data back, but we want to make sure you have a choice and can do what works best for you.
Please let us know as soon as possible which option you would prefer so that we can help to restore your account as quickly as possible.
Again, we sincerely apologize for the stress and inconveniences that this series of events has caused. It has been a stressful few days for everyone, but we are delighted to let you know about the progress we have made.
As always, if there is anything that we can do to help you or any further questions we can answer, please do not hesitate to let us know.
--
Jen Lepp
Director of Operations
A Small Orange
|
|
25th January 2012, 10:45
