Digital Ocean completely destroyed my droplet beyond recovery out of the blue

Posted by phaser on Tuesday August 26, 2014 @06:13PM
from the self-post dept.

  From: DigitalOcean <[email protected]>
  Subject: [DigitalOcean Ticket ID:xxxxx] cant power on droplet
  Date: August 13, 2014 at 1:26:58 AM GMT-4
  To: <xxxxxx@xxxxxxx>
  Reply-To: <[email protected]>

There has been a response to your ticket:

Hello,

You will be unable to power on this droplet as it has suffered unrepairable damage.

We are writing to let you know that the hypervisor that hosts (redacted) has suffered a catastrophic failure. Despite repeated attempts to replace failed components and even to perform two full swaps of the system chassis, we were unable to recover it.

Unfortunately, the failure also resulted in loss of all data on the hypervisor. Droplets and data hosted on this hypervisor node are not recoverable, despite all of our efforts.

Please let us know if you need help recovering from a recent snapshot or backup. While we know that an account credit is only a small comfort when confronted with data loss, we have gone ahead and credited your account for one month of service.

If there is anything else we can do, please let us know. We will be standing by to assist you in recovering from this as best we can, and we have marked the IP that was associated with this Droplet as reserved to your account, so your next Droplet in the SFO datacenter should reclaim it.

Regards,

(redacted)

(edit: line breaks)

( 14 comments | 12 points | on HN )

Re:Digital Ocean completely destroyed my droplet beyond reco... by Artemis2 on Tuesday August 26, 2014 @07:48PM (#8230212)

It's the cloud. You have backups and you can spin up a replacement server in a minute. If you don't have backups, you should probably rethink the way you use these services. Hardware fails, inevitably.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by wmf on Tuesday August 26, 2014 @06:26PM (#8229925)

So restore from backup. Hardware failures are inevitable.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by mariust on Tuesday August 26, 2014 @06:39PM (#8229974)

I do think that, hardware failures are inevitable, but using a RAID system, should prevent such things since all data is written on 2 or more disks at the same time. The backup might be one or more days old, think of Facebook would say: " hey 1 billion users the 10x billions of images you have uploaded in the last 24h are gone, please upload them again "

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by TheLoneWolfling on Tuesday August 26, 2014 @07:02PM (#8230061)

RAID is much less effective than it used to be.

Current BER (non recoverable read error rate) is around ~1/10^14.

If you lose a drive, the probability of you encountering an error when rebuilding the RAID is rapidly approaching 1 as time goes on - BERs are not decreasing nearly as fast as drive sizes are increasing.

(This assumes you are using a RAID where you can lose 1 drive and be fine. RAIDs where you can lose 2+ drives are still fine for now.)

And this isn't taking into account catastrophic controller failure, damage due to a faulty power supply / surge, etc.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by jlawer on Tuesday August 26, 2014 @07:01PM (#8230056)

RAID only works when the RAID controller doesn't screw up, or critical metadata doesn't get wiped. If the raid controller starts being the problem your data goes with it. Problem can also happen if your hypervisors filesystem / storage drivers go screwy, get corrupted or one of a million other problems.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by phaser on Tuesday August 26, 2014 @07:46PM (#8230207)

i did. i posted this as a remainder for people to do their backups, especially under digitalocean

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by cypher543 on Tuesday August 26, 2014 @07:52PM (#8230225)

The title makes it sound like you are accusing Digital Ocean of destroying your droplet on purpose for no reason.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by phaser on Tuesday August 26, 2014 @09:28PM (#8230484)

Probably you're right. My writing skills are not the best, but the real intention was to point out that this things happen

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by jordsmi on Tuesday August 26, 2014 @08:05PM (#8230256)

They knew man

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by trampish on Wednesday August 27, 2014 @03:14AM (#8231105)

Did you have DO's backup service enabled when this happened? If so, I feel like they should have provided a more streamlined customer experience restoring the damaged/lost droplet from the latest Amazon Glacier instance.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by mattkrea on Tuesday August 26, 2014 @08:27PM (#8230324)

When backups can be enabled at 20% of instance cost I'll deal with it. Nothing beyond development work runs on only one instance due to the problems I've seen even on Amazon.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by ksec on Wednesday August 27, 2014 @05:41AM (#8231387)

That is why you should be on Linode is you are in production.

Note: Not saying they are perfect, but it is just much less likely to happen.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by farawayea on Tuesday August 26, 2014 @06:54PM (#8230032)

They are using RAID 5 and probably cheap ssd storage. I'm not surprised. This is what they give you for your money.

[ Parent | on HN ]

Re:Digital Ocean completely destroyed my droplet beyond reco... by tuananh on Tuesday August 26, 2014 @09:29PM (#8230486)

happened to me once. i wonder what's the actual SSD failure rate at DO? Is it any higher than industry standard rate?

[ Parent | on HN ]