It has nothing to do with 'not invented here', it has everything to do with your inability to outsource your responsibilities.

The degree to which people rely on others to take care of their stuff is a huge blind-spot. Jason's advice is spot on in this respect, no matter what the up-time guarantees of the cloud solution you are using (and no matter what the redundancies), if you store all your data in the cloud without an off-line copy your company is 3 mouse clicks away from being history.

3 copies

2 formats

1 copy off-site

The cloud is a great place for that 1 off-site copy.

The vast majority of the world would be much better served by (to appropriate your jargon) a 2-1-1 rule, because it's a straightforward treatment for a straightforward problem that is easily implemented. Yes, "format skew" and double-failure of backup solutions does indeed happen, but at a much lower incidence than "oh crap I deleted it!".

This protects against data corruption that is not detected immediately.

Another common way of doing this (and one that I prefer) is one where you rotate out a backup medium with ever larger intervals. So one gets set aside per week, then one gets set aside per month and so on. That gives you a series of snapshots in time that will allow you to pinpoint with some accuracy when an event happened. Longer ago you'll have less accuracy but this can help a lot in trying to triangulate who or what messed up. Just being able to answer the question of whether or not 'x' happened before 'y' was hired or after can help in narrowing down the number of suspects in case of a breach or other nastiness. It also prevents against back-ups for whatever reason not wanting to be reloaded (and you should guard against that by loading your back-up immediately after you make it, even so, the medium might fail the next time you try a read).

Better still if there is a streaming log of everything but only very few companies can afford that sort of solution for all their data. Those can be hard to restore from (by replaying) so there too a snapshot system can help.

But if you're going to condense it to a "rule" that will help people not well-versed in the field, that rule can only be "MAKE BACKUPS!", because at least 90% of the data loss scenarios in the real world happen because simple backups weren't made.

Don't make it more complicated than it is, because someone will stop to do it "right" and then lose data because they didn't just make a copy on a USB stick.

3 backup services 2 formats 1 local copy

I guess that could get expensive depending on the services.

For each business this is a delicate affair and you should spend some time on figuring out what you really absolutely have to have and what you could lose without losing the company or getting in trouble with the law.

There's risks and then there's risks. As always, you can spend some time and money to mitigate some risk, and decide some risk is low enough that you don't care to mitigate it. I wouldn't just assume one risk vector is higher than another without some concrete numbers.

What are the risks of having a bad health issue happen to you? Are you insured?

I'll bet that you don't know the answers to any of those questions, and yet, you are probably insured against all of them.

As for your question in more detail: it happens often enough that for me if you operate your business 'in the cloud' and you don't have a back-up of your data outside of the cloud to guard against catastrophic data loss due to either malice or accident that I'll happily fail you. Think 'sysadmin was depressive, wiped out company' (or maybe you let him/her go and they took revenge or any one of a number of other scenarios that would instantly terminate your existence online if you had not guarded against it).

Risk management is consequences * incidence versus cost to mitigate. If the cost to mitigate is negligible, the risk is measureable and the consequences are terminal then it is a no-brainer to protect yourself against this. At least one of my customers wished they had set up a backup facility for their critical data (too bad, end of story for them) and there is one published case that we all know about. That's two that you can count on and most likely other trouble shooters have similar stories.

I see catastrophic cloud data loss as having a much higher chance of happening than many other items on the list of stuff that I verify before greenlighting an investment.

Though if we unbox that a bit... How do we balance the risk of loss due to your cloud being owned vs. loss due to the risk of your users' PII being violated if someone attacks your in-house (ostensibly in-house reliability and security-managed) copies? Better make sure you're spending the money on security best-practices on all your copies, not just the "live" ones.

Side-note: What are your thoughts on two separate cloud providers as providing sufficient insurance? Say, having your data live in Google Cloud and at rest in Amazon S3?

Having two cloud providers would work assuming they don't share any critical infra-structure and assuming that it is not the same set of employees that have access to both systems. Also helps if you have a totally separate set of credentials for the back-up system and if that back-up system can only be unlocked by very senior people (preferably execs) after a catastrophe of suitable magnitude hits.

Whether you store your primary in the cloud or your back-up in the cloud the story is the same: have a copy somewhere else.

Finally: the only real back-up is one that is off-line. So from that (ok, ultra-paranoid) viewpoint it would be best if you actually went to an off-line medium to store your data in such a way that nobody can wipe it all out without physically destroying all copies.

All this of course after suitably weighing the importance of the data.

"Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in an irreversible position both financially and in terms of ongoing credibility. As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us."

They promised a follow up once they got to the bottom of what happened but they never did, I'm still curious what the whole story was.

I can make this even easier. Its almost certainly going to boil down to this question. Have you ever lost even a single byte of data? Cause if you have, you aren't even in the same ballpark.

Have you ever been killed in a car accident? No? Then I guess you don't need a seatbelt, right? Has your house been burnt down by a lightning strike? No? Then no lightning rods and proper grounding for you, right?

You don't wait until a disaster occurs before you put a disaster recovery plan in place.

So getting back to risk management. So we about to make a QB selection for the big game. Now that we understand that the implementation of folks like jaque here are like the 12 year old pee wee standout vs. the cloud provider's seasoned NFL quarterback. Which one do you pick to safe guard your business.

This is just alarmism. If you really wanted to demonstrate your point you would show me some data. The data would demonstrate that over the millions and millions of users on a number of cloud platforms that their rates of data loss are significantly higher than your "home spun" storage. Then you would take out the outliers and show an honest distribution on the most stable vendors(since I'd expect the majority of data loss over all vendors to have some amount of locality within a specific vendor). The result of this will be the rate of data loss on the most reliable cloud platforms. You can then compare this against your own success rates.

If you have ever(even once) lost even a single byte of data(for any reason), I do believe you will find yourself hopelessly outmatched.

You seem to be missing the point.

People are not saying "don't use cloud storage at all"

They are saying "don't use cloud storage exclusively"

You might trust that your cloud providers will never be hacked, that your individual account on them will never be hacked, that they will never suffer a catastrophic failure, that they will never go out of business, that they will never be fully or intermittently down at the moment you need to retrieve some data, and so on and so forth, based on their claims and documentation, or that you won't cock up at some point and cause data loss in your account, but I prefer to have my own copy (or copies) as well as the ones that are "in the cloud". For truly essential data at least one of those copies is both offline and offsite.

For most values of "I have x hours and y dollars to safely store z gigabytes", a pure-cloud solution (possibly involving multiple independent cloud providers) has a lower chance of failure than one involving local storage.

But the fact remains that where ever you choose to put it "offline and offsite" the odds of it being lost are orders of magnitude greater then its persistent and redundant storage on a reputable cloud provider. Even if you put it on the most stable storage you can find and lock it in a underground safe. You can't guarantee its integrity a handful of years from now let alone centuries.

To put it in other words you are advocating storing your money in a mattress because you don't "trust the banks"

It is interesting that even today there are still people arguing against having back-ups because 'the cloud', you'd think that we ITers learn from our mistakes but for some this has to be personal experience before the lessons are learned. Good luck if and when it happens to you, please re-read your comments here at that time. There isn't an IT person that I know that has not been saved by a back-up at some point in their career, the fact that you are lucky so far is not a reason to think you are exempt.

And no, nobody is advocating storing your money in a mattress because you 'don't trust the banks', the point is that you can keep a copy of your critical data for very little money to safeguard against an eventuality that seems to hit the IT world with alarming regularity, even if they use 'the cloud'. Fuck-ups, hacks, disgruntled employees and failures all happen, every day.

The failure probabilities are relative. And everyone here is equivocating. As if the likelihood of failure is evenly dispersed. You indicate that storing customer data in the cloud is "playing fast and loose" but I'd argue the opposite. Not having a cloud backup is what is "fast and loose"

Imagine you are an independent bank. You need to move your customers deposits. Do you load sacks of cash into your corporate minivan or hire an armored car service? Well lets look closer. With the latter you are giving up control, right? You have no control over the quality measures an armored car service might take. Yet somehow not contracting them seems like foolishness. The reason is obvious. Its because you know in this case that transporting money isn't your expertise. Its not something you can focus adequate time and resources on perfecting. You also can't spread the risk of failure across a lot of customers, absorb that failure, and make your service better for the next go. The exact same logic applies to long term storage of data. If that isn't your only function its extremely hard to get it right.

If you bank fails, you will be made whole with money from the FDIC. Money is fungible. Any money will do.

Data is not similarly fungible. There's no IT FDIC to replace your lost data with other data that's identical.

When was the last time the FDIC failed versus the last time an online storage vendor folded?

Personally I think the risk that AWS fails is remote, but if I were to store a bunch of data with them I'd most definitely make sure that I would not be dependent on them. It would be a convenience at best, but never a dependency.

> You indicate that storing customer data in the cloud is "playing fast and loose" but I'd argue the opposite. Not having a cloud backup is what is "fast and loose"

Basic reading comprehension failure, that is not what GP is arguing.

He's arguing that if your live data lives in the cloud your backup copy should not be in the cloud and vice versa.

Insured by who? Not governments. Governments collapse all the time. The only safe place to keep it is in your own custom bank at home that you built yourself.

Money and data are not directly compatible here, as one "bit" of money is the same as any other in most respects, but I do always carry a minimum amount of cash in case my cards stop working or are lost/stolen, have a pot of "emergency cash" at home for similar reasons, and recommend others do the same. Most of my money is in the bank so I'm not protecting against massive institutional failure here, but I am protecting myself against a variety of possible temporary inconveniences and possible mistakes on my part.

I'd rather have two backups of which one is more likely to fail then the other than just one, or three rather than two, and mixing types of backups means not all of them are subject to exactly the same failure modes (my local backups, online or off, are unaffected by connectivity issues for instance). For data that is important, a little paranoia is healthy IMO.

In this case, money can't be compared to digital data. You can make multiple copies of digital data on our own but not of your money tucked away into a mattress, that would be illegal.

I also not advocating against the cloud, just that a single copy in S3 is not a backup solution.

There is no single perfect solution, so like security you layer your disaster recovery.

This is more or less an exact copy of how I would advise companies to set this up.

(Jason Scott is speaking from experience here, as one of the people called in to do emergency archive response when these cloud businesses shut down.)

Edit: jacquesm points out the sudden death of cloudspaces.com, which I'd forgotten about: http://www.infoworld.com/article/2608076/data-center/murder-...

A single computer security problem (which happen on a daily basis, although not usually at this severity) could enable your cloud to be deleted.

The "counterparty risk" is small but ineradicable. The finance industry re-learnt this with Bear Stearns recently.

That's absurdly unlikely to happen with S3.

I'm not going to go too deeply into speculative scenarios, but all kinds of business-interrupting calamities are possible with very low probability. We just had the story on the frontpage about the Juniper backdoor; do you think S3 isn't being targeted by multiple state intelligence agencies?

Basically I agree with https://news.ycombinator.com/item?id=10772321 - by all means keep the working copy online, but have a local offline backup.

How absurd was it that money market funds would fall below $1 during the recent unpleasantness ? It was so unthinkable that major parts of our entire global economy were predicated upon it never happening.

It happened.

Money markets are even more "serious business" than Amazon, or even all of tech is, and had even more big brains attesting to their inability to fail. They failed.

If Amazon went out of business there is no chance at this point that the loss would be total and catastrophic. Its essentially a bank at this point. Its shutdown would be orderly.

For any one system to fail is perfectly possible and in fact should be expected to happen at some point and that's why you design against that.

You also make sure that it is not the same people that have access to both systems so that if your sysadmin walks onto the floor with a bad hairday your back-ups will still be there. And this is also why you test your back-ups to make sure they actually work.

The cost of getting stuff out of Amazon is very dear. While the statistical likelihood that Amazon will lose your stuff is low, the likelihood of Amazon or a competitor doing something to cause you to rethink your use of AWS (or force you to move) is much higher. Wanting to reduce your AWS costs in the long term is a near certainty.

As vendors move away from perpetually licensed software this gets more important. What happens when a vendor (say Microsoft for strawman purposes) decides that some function that is important to you must interface with Azure, and Azure only? What if Amazon decides upstream network transfers are no longer free? Those sorts of changes can break your business, and vendors like Amazon/Microsoft/Google are fully enabled to change those terms.

I think there is an easy rebuttal that should be considered...

First, the "nines" rating of any service or resiliency is just gibberish. Go find the statistical likelihood of money market funds "breaking the buck" or of CDS blowing up - both in 2007/2008. Those had a lot of nines too and a lot of very smart , well qualified people attesting to those nines (in venues even more serious than IT).

A highly complex system becomes incomprehensible, even to the people that built it. Those nines mean nothing.

Second, you absolutely can build something more stable and predictable than Amazon precisely because you're the one that built it - which means that it is more comprehensible and fails more predictably and gracefully.

I don't care who does the calculation and how many nines they come up with - if you load FreeBSD on two bare metal servers and put them in two different datacenters and run them with any kind of conservative and cautious sysadminning you'll have a better solution. Yes, it will be more expensive.[1][2]

The standard closure to a comment like this is to refer to Talebs Black Swan and Antifragile books ... which you certainly should read ... but even more important is "Normal Accidents" by Charles Perrow[3] which I hope will convince you to stop looking for complex things that never fail, and instead look for simple things that fail gracefully.

[1] ... but we have a HN-Readers discount - just ask!

[2] You know who we are.

[3] https://en.wikipedia.org/wiki/Normal_Accidents

Nonsense. It's very easy to have that kind of setup fail - remove one from the load balancer, take it down for maintenance, whoops we removed the wrong one from the load balancer. I've been in similar-sized companies using dedicated servers or AWS, so I've seen both sides. It's like how people feel safer when driving themselves than when being driven by a professional, even though a professional is overwhelmingly likely to be a better driver - everyone thinks "oh, there's no way I'd make that kind of mistake".

It is very easy to overengineer "high availability" systems - I certainly think things like STONITH and dedicated control planes are more trouble than they're worth, and running a system that you don't understand is a recipe for failure. But I'll take FreeBSD on a basic AWS setup - four EC2 nodes (multiple AZs), and an ELB - over bare metal any day. Physical server maintenance is not where I have a competitive advantage, and would mean more complexity to understand, not less.

I'm sorry - you're already missing the point.

There is no load balancer. There's no firewall. There's no services running except for sshd.

I guess I should qualify what I mean by "better", though. What I mean is, "I know exactly how this will fail and it won't be interesting or surprising. Or take any thought or time to fix."

It will be comprehensible. "FreeBSD on a basic AWS setup - four EC2 nodes (multiple AZs), and an ELB" is not. You have no idea (nor do I, nor do probably most folks at Amazon) the different and fascinating ways that will fail.

(disclosure: we use EC2 instances for backup DNS. We are not anti cloud or anti amazon)

This is from you and this is from the wikipedia article on NIH(not invented here) syndrome ...

>Not invented here (NIH) is the philosophical principle of not using third party solutions to a problem because of their external origins. False pride often drives an enterprise to use less-than-perfect invention in order to save face by ignoring, boycotting, or otherwise refusing to use or incorporate obviously superior solutions by others.

I am not saying don't have a copy, I am saying the if you have several copies the safe one is in the cloud.

These polarized "requirements" are usually rationalized away by each group. As a data privacy advocate, I believe that I can provide a reliable storage solution to my company without using a centralized cloud service, which then guarantees my privacy because "I'm in control". As an advocate of centralized cloud services, such as Amazon, I present that their team is better at security and reliability than any other team on the planet and that I can encrypt something and trust that my key management is secure. Both of these arguments have fallacies and assumptions.

The solution is to challenge ourselves to build better solutions. At what point in time did technology advancement ever slow down? At what point will we ever stop and say "Y'all, this here compute system is good enough and should be centralized/decentralized!"? Never, I say.

    and if you’re a person they are giving it to you without
    you signing anything accompanied by cash or payment that
    says “and I mean it“.

For plenty of use cases I actually agree with that conclusion. With the caveat that I would add: accompanied by a suitable non-cloud back-up of critical data, code and configuration data held on a medium that is not accessible by the same people that administer the cloud stuff.

This is where you are simply wrong.

> "the authoritative copy goes into the cloud"

This we agree on.

So, where you are wrong is this: you are only considering technical failure modes but there are many others besides and so any comparison between 'the cloud' and 'your get-out-jail-free-card-backup' would have to be in terms taking into account all modes of failure, not just the technical ones and then on top of that you'd have to consider the likelihood of the backup failing to restore at the same time that the primary system goes down for whatever reason (including technical ones).

That's why I call it a blind spot, even after being told in 20 different ways that the uptime of Amazon does not come into play here you are still clinging to that. It is simply not relevant because that's not the scenario a back-up will most likely prevent against. It will also prevent against that scenario, on top of the ones where someone incapacitated by drugs, anger, depression or any one of another set of circumstances decides to take it out on your company. Or maybe someone makes an honest mistake (I've been called out twice in my career to restore data for companies that had been wiped out because of simple mistakes that should have never ever happened).

So back-ups are not a luxury, they are a necessity and the degree to which Amazon can outperform your back-up in terms of availability is not a factor in the whole discussion.

Ask Bernie Sanders about the cloud and the implications of losing access to it at the whim of the cloud provider.

Now go put your stuff up on S3 but please keep a backup of the things you upload there.

>Do you control Amazon? No? Then you're not the captain

This is textbook NIH syndrome. Rather than looking at the relative probabilities you look at something from an ideological perspective. Your argument isn't "Its safer with me" its "I want to be in control of it" This attitude is generally harmful. Think about your money. Are you in control of the bank? no right, but you still keep your money there instead of in a shoe box under your bed. Why? Because the bank is better at protecting your money than you are. They have big heavy metal doors and men with guns to move it from place to place. Amazon is like a bank for your data.

I'm not saying you shouldn't use S3. You should just make sure that it's not the only service you have the only copy of your data hosted.

>Rather than looking at the relative probabilities you look at something from an ideological perspective.

No, I'm looking here solely at the risk of putting all your eggs into a basket you don't control.

That is what happened (once again) to John Meriwether and Long-Term Capital Management... A "once every 10.000 years event" which took place... today.

Where things like that in 2009 when this article was written?

Curious if Upton Sinclair would have anything to do with it.

Lets put it this way, I know enough not to store data on the cloud(s) exclusively.