News for hackers. Stuff that matters.
Re:DigitalOcean lost our server by mizzao on Thursday February 04, 2016 @10:31PM (#11039219)
I'm more worried about the following post in that thread:

<begin quote> --------------------------------------------------------

We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No biggie, we'll just update it, turn the server back on and be on our way.

Nope. We had to contact customer service to get back in to our account. After updating the card info we realize that all our droplets are gone. We reach out to customer service again in which case they let us know that when a CC expires, an automated process kicks off and deletes the droplets. We've been a customer for 2 years, surely they could pick up the phone and call us. We've spent thousands of dollars with them.

Plan B, let's restore them from the backups we've been paying for. Nope! When they delete your droplets, they also delete your backups.

This is where we ask to speak to them on the phone and are denied. I then ask if they are insane and why they would delete someone's servers and backups via an automated process without a human at least checking to see if it is a loyal multi year customer who has a simple lapse with their CC expiry.

We had to find a backup on a developers machine from over a month prior and rebuild data using various megtods that took close to a week. In the end, DI gave us a $500 credit.

You get what you pay for. Anyone who uses DI for production or anything more than a hobby app is playing with fire. They do not care, they are apathetic, they will screw you over and the throw you a credit for their terrible service as a half assed apology.

Re:DigitalOcean lost our server by mizzao on Thursday February 04, 2016 @10:39PM (#11039241)
One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day.

What if I was on vacation, and didn't see this e-mail for a few days? shudder

In what world is automating the deletion of customer data a good idea?

Re:DigitalOcean lost our server by dualboot on Thursday February 04, 2016 @11:32PM (#11039458)
> In what world is automating the deletion of customer data a good idea?

The data destruction world :)

Re:DigitalOcean lost our server by xyzzy4 on Friday February 05, 2016 @01:15AM (#11039856)
The world where employees care more about management satisfaction than customer satisfaction? And it's difficult to prove that bothering some customers here and there is impacting corporate revenue. Policies like this slip through the cracks.
Re:DigitalOcean lost our server by fleitz on Friday February 05, 2016 @01:42AM (#11039965)
Yes, it's digital oceans fault that people want to pay bargain basement prices and then wonder why they are getting bargain basement service.
Re:DigitalOcean lost our server by mixmastamyk on Friday February 05, 2016 @08:15PM (#11046005)
Sorry, that doesn't excuse the purely bad idea of scripting deletion of data at the first sign of trouble.
Re:DigitalOcean lost our server by Tassels on Friday February 05, 2016 @09:31PM (#11046231)
It's not at the first sign of trouble, though. It's after ~15 email notifications over the course of over a month, and no contact back.
Re:DigitalOcean lost our server by mixmastamyk on Sunday February 07, 2016 @03:01PM (#11054526)
Both anecdotes mentioned timeframes of about 15 days after longer than a year of service.
Re:DigitalOcean lost our server by Tassels on Monday February 08, 2016 @09:45PM (#11062717)
The timeframe is 3 weeks until suspension, then another 2 weeks until deletion. At the time this user was affected, it was 3 weeks until suspension, then 3 days until deletion, but that period was raised to 2 weeks due to feedback that 3 days wasn't enough.

I'm not saying that excuses things, but it's far from "at the first sign of trouble".

Re:DigitalOcean lost our server by the_common_man on Friday February 05, 2016 @12:29AM (#11039706)
I had the exact same experience! Except I got no credit.

My CC expired but I was on vacation and didn't have access to email. When I was back, the droplet and all backups and snapshots were completely removed. For the record, I was only 15 days late. Very sad. I tried my best to get the backups from the support engineer but no luck.

Re:DigitalOcean lost our server by mrweasel on Friday February 05, 2016 @02:50AM (#11040165)
>We've been a customer for 2 years, surely they could pick up the phone and call us.

I don't understand that reasoning from customers to be honest. That right there, throws away the $5 and $10 plans. Having some one trying to contact a customer, just because their credit card expired, cost more than the actual hosting.

Top tier service does comes from companies with $5 a month plans. I'm sorry it just doesn't.

Re:DigitalOcean lost our server by josephagoss on Friday February 05, 2016 @03:47AM (#11040327)
> We've spent thousands of dollars with them.

That's a decent amount of money right? It sounds like they were not using the $5 plans.

Re:DigitalOcean lost our server by mrweasel on Friday February 05, 2016 @05:26AM (#11040593)
Over two years, that's what, eight $10 droplets or sixteen $5 droplet.

I would guess that if you want a hosting company to be proactive you need $100+ per month.

Re:DigitalOcean lost our server by aagha on Saturday February 06, 2016 @02:23AM (#11046904)
Because $5/$10/month is not the same thing as Customer Lifetime Value.
Re:DigitalOcean lost our server by josephagoss on Friday February 05, 2016 @03:40AM (#11040307)
Thank you for this, I have a large client where we were considering DigitalOcean and the yearly spend could easily be in the thousands. Now I'll strike them off of the list. Automated deletion of VM's and backups, absolutely unacceptable.

The VM hosting market isn't looking good at the moment, seems like each provider has some dark history at this point, it's really frustrating.

Re:DigitalOcean lost our server by rorykoehler on Friday February 05, 2016 @01:32AM (#11039937)
For what it's worth this happened to me through a client's account (client wasn't checking emails and didn't see warning mails re CC expiring). I reached out to customer support once the site went down and even though we were beyond the cut-off deadline they re-activated all the droplets. Perhaps they have updated their process in this regard. I also learned my lesson and made sure I got forwarding on all mails related to DO.
Re:DigitalOcean lost our server by andrewthetechie on Friday February 05, 2016 @05:10PM (#11045026)
Check out the followup:

<begin quote> Hey Matthew,

I'm Zach, Director of Support here at DigitalOcean. Thanks for raising this topic.

I was able to locate your account, and I see that I was the one who granted the credit and followed up via email. I hadn't heard back from you until now, and I'm happy that we have an open line of communication. I'm hopeful that this thread starts a conversation, that it clarifies what steps we take, and we might even uncover a different solution that works well for everyone.

To start with, I want to be really clear, this is the absolute worst case scenario. I absolutely don't want it to happen to any customer, let alone someone who has been with us for such a long time. There's really a delicate balance that we must strike between customers who forget to pay and customers who do not want to pay.

Currently, our notifications for overdue balances are sent via email. If a customer account is unpaid on the first of the month, we send ~15 emails total notifying the customer of the situation, and subsequently power off servers 21 days after the account is on hold as a further way to gain your attention. At this time, droplets are removed from your account 14 days following power off, which is an increased amount of time from what you experienced. It was increased from 3 days based on past user feedback.

Why do we do it this way? In the past, we had no hold or suspension process for non-payment, which enabled bad actors to run for months and months without paying. From a business perspective, we made a decision to put a scalable process in place that limited how long an account could go unpaid.

As a support team and business, we are always willing to work with our customers who are unable to pay. We are always available via ticket, our contact form, and have made wide-ranging attempts to help users who aren't able to pay due to banking regulations: https://insidedigitalocean.com...

I'd like some input, so I'll publicly ask for feedback on questions that I've asked privately before. I'd also like any other feedback that we can consider on how to make this a more positive experience for everyone

-Knowing that we do not do phone support, what's the best way to notify you of a past-due balance? -Is SMS effective at times like this?

I would love to hear your thoughts.

Thank you, Zach [email protected]

Re:DigitalOcean lost our server by mizzao on Monday February 08, 2016 @01:32PM (#11059939)
3 days seemed like an accident waiting to happen.

This updated policy seems a lot better; it would probably be great to ask people for secondary contact information such as SMS for emergencies.

Re:DigitalOcean lost our server by toadi on Friday February 05, 2016 @03:49AM (#11040335)
Offsite backups? We use AWS and DO to host applications for our clients. We still take offsite backups. Isn't that what your clients paying you for?
Re:DigitalOcean lost our server by vidyesh on Friday February 05, 2016 @01:01AM (#11039810)
I really don't know to admire or criticise DO for this.

On one hand horrible situation to be in, at least could have called/mailed/SMSes/notified the customer before nuking the account.

On the other hand, it is a relief to know your data is secure and gone one you stop paying, no way they have a copy or keep it in archive/deep backup.

Re:DigitalOcean lost our server by happywolf on Friday February 05, 2016 @01:21AM (#11039886)
Nope, the only assurance you can get is you won't be access your data, I am not sure how one can come down with conclusion of "no way they have a copy or keep it in archive/deep backup"

Having a backup and letting you to access it are two entirely different issues, and most of the time customer service staff don't know how the data management is being done by the engineering/operations team.

Re:DigitalOcean lost our server by vidyesh on Friday February 05, 2016 @01:33AM (#11039938)
If you have used shared hosting you know as soon as you pay for your expired account, everything is back as it was. Never thought this was a problem until you try to test how long you can go until you pay and still restore you data.

If DO's CC can't handle data management problems they probably will connect you to their engg/ops team. So I would assume they would prefer having it resolved ( by giving the data if they had ) instead of keeping it and let user rant/post about it( which they know will probably happen ).

Re:DigitalOcean lost our server by happywolf on Friday February 05, 2016 @06:01AM (#11040700)
I agree with the first paragraph.

The second paragraph hinges on the rationale that if customer data were available, then CS (Customer Service) agents would have made every effort to contact the ops/engineering team to retrieve those to avoid a social disaster.

Unfortunately, this isn't true in the companies that I worked for, which are big customer-facing MNCs.

First of all, for US consumers, the CS teams most likely are contractors in Philippines, Malaysia, or India, they are trained with scripted responses. For any issue that is beyond the scripts, their standard response will be "Sorry we can't help you because we can't do x", one reason is they have no incentive to escalate the issue and get oneself more work to follow up, other times, it is the management explicitly decided this is to prevent too many cases hit the ops and eventually, the engineering teams. It is not that the management is stupid or nasty, just that the 20:80 rule dictates 80% of the issues usually are raised by 20% of particularly picky customers. For the 20% customers who generate 80% of the revenue, each of them will be assigned an AM (Account Manager) who can get things done much faster and more efficient than the CS team. This group of high-value customers are often given access to VPs to make sure all their issues are heard and handled properly

Second reason is most teams in a big company will not concern anything that doesn't hit them directly. For example the CS team in my ex-company upon getting repeated complaint on an erratic component that didn't get bug fix for months, eventually the CS told the upset customer to fuck off and complain to CEO directly since it was engineering department's fault.

Unbelievable? Maybe, and that company remains one of the largest high-tech companies in the world.

Re:DigitalOcean lost our server by Tassels on Friday February 05, 2016 @10:03PM (#11046338)
For the record, that's not how it works at DO. The support department encompasses Platform Support (like T1), Trust & Safety (handles cases of possible fraud or abuse complaints), and CloudOps (sysadmins who help maintain the cloud).

I'm on Platform Support and while I can't access the actual hypervisor hardware like CloudOps can, we do not have "scripts", and we always do our best to help customers in any way we can. We also have a myriad of tools that we can use to monitor our platform and help troubleshoot any issues that may arise in our tickets.

My roommate and bestie is on CloudOps, and she sits right next to me at our HQ in NYC. While we have a great remote culture, all of us on support are in the US, with the exception of one guy in London (who started last week and is AWESOME).

Not only that, but I work very closely with all of the other departments here at DO, and our executives are extremely accessible as well. If we need to pass on feedback, we always do so.

We're also encouraged to go out of scope and do whatever possible to help out our customers, and I wouldn't change a thing. I love what I do, I love the people I help, and I love my coworkers.

Re:DigitalOcean lost our server by josephmx on Friday February 05, 2016 @05:10AM (#11040546)
I had the opposite experience, my CC expired with DO, but I didn't have a new one for about 2 weeks. I didn't have any issues with just waiting to the next billing cycle and paying double then to cover the negative.
Re:DigitalOcean lost our server by fweespeech on Friday February 05, 2016 @03:44PM (#11044468)
This is why you use two providers with two credit cards.

I'm quite serious. Our production environment operates on two cards with different issue & expiration dates from two different merchants and uses 2 different Cloud providers.

Re:DigitalOcean lost our server by techstrategist on Friday February 05, 2016 @03:28AM (#11040277)
This is a big danger of "move fast and break things". I imagine the process they put in place seemed reasonable internally before catching this edge case, but what a crappy result.
Re:DigitalOcean lost our server by joenathan on Friday February 05, 2016 @01:19AM (#11039880)
I'm looking for a good host, was considering DO, but now I'm not. Any suggestions?
Re:DigitalOcean lost our server by joshmn on Friday February 05, 2016 @02:04AM (#11040034)
Needs vary, but here's a brief list of competitors:

* OVH.com — huge, great price point and performance

* Linode.com — also huge, favorite of many developers

* Vultr.com — owned by Choopa (bigger player)

* RamNode.com — doesn't scream "super modern", but reviews are great

There are many others, but those are off the top of my head that correlate with DO.

Re:DigitalOcean lost our server by jacquesm on Wednesday February 10, 2016 @01:55PM (#11075019)
- leaseweb

- 1andone

Vote against linode here, yes, they're large but they are also not all that good.

Re:DigitalOcean lost our server by toadi on Friday February 05, 2016 @03:51AM (#11040341)
According to me DO is still a good hosting company. Works good for the price you pay. Taking offsite backups is always a good idea.