News for hackers. Stuff that matters.
Re:"DigitalOcean Killed Our Company" by _bxg1 on Friday May 31, 2019 @04:34PM (#20064732)
It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.
Re:"DigitalOcean Killed Our Company" by stingraycharles on Friday May 31, 2019 @04:46PM (#20064818)
More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.
Re:"DigitalOcean Killed Our Company" by jdietrich on Friday May 31, 2019 @05:19PM (#20065070)
>getting locked out of your account is usually just not a consideration

How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan.

Dumping everything to B2 is trivially easy, trivially cheap and gives you substantial protection against total data loss. It also gives you a workable plan for scenarios that might cause a major outage like "we got cut off because of a billing snafu" or "the CTO lost his YubiKey".

Re:"DigitalOcean Killed Our Company" by williamdclt on Friday May 31, 2019 @06:05PM (#20065435)
> How many horror stories need to reach the front page of HN before people stop believing this

Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"

Re:"DigitalOcean Killed Our Company" by kelnos on Friday May 31, 2019 @06:41PM (#20065729)
Even if the stories we hear of account lockouts isn't typical, the absolute number of them that we see -- especially those (like this one) that appear to be locked (and re-locked) by automated processes -- should be cause for concern when setting up a new business on someone else's infrastructure.
Re:"DigitalOcean Killed Our Company" by jdietrich on Friday May 31, 2019 @06:58PM (#20065842)
If you plan for the "all of our cloud infrastructure has failed simultaneously and irreparably" scenario, you get a whole bunch of other disaster scenarios bundled in for free.
Re:"DigitalOcean Killed Our Company" by _bxg1 on Friday May 31, 2019 @04:52PM (#20064860)
Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup.

Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.

Re:"DigitalOcean Killed Our Company" by yourbandsucks on Friday May 31, 2019 @05:07PM (#20064969)
And there are a million stories of startups who build the wrong thing, don't achieve product-market fit, etc.

You can't dot every I, cross every t and also build a compelling product as a 2 person shop.

Re:"DigitalOcean Killed Our Company" by andrewflnr on Friday May 31, 2019 @05:36PM (#20065216)
Backups aren't "dotting i's and crossing t's", they're fundamental. FFS, just rsync your database directory somewhere.
Re:"DigitalOcean Killed Our Company" by icebraining on Friday May 31, 2019 @05:55PM (#20065363)
Then maybe you shouldn't be building that product with a 2 person shop.
Re:"DigitalOcean Killed Our Company" by yourbandsucks on Friday May 31, 2019 @10:04PM (#20066695)
Sounds good in theory.
Re:"DigitalOcean Killed Our Company" by ufmace on Friday May 31, 2019 @06:52PM (#20065806)
I don't know, it seems simple enough to me. I have a server on DO hosting some toy-level projects, and IIRC it took me 15-30 min to set up a daily Cron job to dump the DB, tar it, and send it to S3, with a minimum-privilege account created for the purpose, so that any hacker that got in couldn't corrupt the backups. I'm not a CLI or Linux automation whiz, others could probably do it faster.
Re:"DigitalOcean Killed Our Company" by Lazare on Friday May 31, 2019 @06:10PM (#20065480)
> It’s unrealistic to expect this of a tiny startup.

I could not disagree more. There's a right way and a wrong way to do this, it's trivial to do it right, and the risks of doing it wrong are enormous.

Re:"DigitalOcean Killed Our Company" by bdcravens on Friday May 31, 2019 @06:30PM (#20065642)
> It’s unrealistic to expect this of a tiny startup.

Then it's unrealistic to trust them with your business.

Re:"DigitalOcean Killed Our Company" by thaumaturgy on Friday May 31, 2019 @04:49PM (#20064839)
That's better than nothing, but still not great.

We don't know the structure of their DB and whether failover is important or not, so we don't know if the DB can be reliably pulled as a flat file backup and still have consistent data.

We also don't know how big the dataset is or how often it changes. Sometimes "backup over your home cable connection" just isn't practical.

Cron jobs can (and do) silently fail in all kinds of annoying and idiotic ways.

And as most of us are all too painfully aware, sometimes you make less-than-ideal decisions when faced with a long pipeline of customer bug reports and feature requests, vs. addressing the potential situation that could sink you but has like a 1 in 10,000 chance of happening any given day.

But yes, granted that as a quick stop-gap solution it's better than nothing.

Re:"DigitalOcean Killed Our Company" by cs02rm0 on Friday May 31, 2019 @04:55PM (#20064881)
> We also don't know how big the dataset is or how often it changes.

I'm going to take a stab at small and infrequently.

Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.

Re:"DigitalOcean Killed Our Company" by thaumaturgy on Friday May 31, 2019 @05:02PM (#20064931)
Yeah, probably. But we shouldn't be calling these guys out for not taking the "obvious and simple" solution when we aren't 100% certain that it would actually work. That happens too often on HN, and then sometimes the people involved pop in to explain why it's not so simple, and everyone goes "...oh." Seems like we should learn something from that. I've gone with "don't assume it's as simple as your ego would lead you to believe."
Re:"DigitalOcean Killed Our Company" by _bxg1 on Friday May 31, 2019 @04:55PM (#20064885)
I suggested that solution because everyone is saying "they're only a two-man shop so they don't have the time and money to do things properly". Anyone has the time and money to do the above, and there's a 90% chance that it would save them in a situation like this.

Even if they lost some data, even if the backup silently failed and hadn't been running for two months, it's the difference between a large inconvenience and literally your whole business disappearing.

Re:"DigitalOcean Killed Our Company" by WJW on Friday May 31, 2019 @05:20PM (#20065084)
Sure it could be. Still not enough companies actually do this though...