News for hackers. Stuff that matters.
Re:Canon's cloud platform has lost users' files and can't re... by ibeckermayer on Tuesday August 18, 2020 @09:17AM (#24198178)
What’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?
Re:Canon's cloud platform has lost users' files and can't re... by lukevp on Tuesday August 18, 2020 @09:22AM (#24198217)
Replication is about having data level redundancy to protect from drive failure. Backups are about having point in time snapshots of the system state, and about having them tiered from a location perspective. The 3-2-1[1] principle says to have 3 total copies, 2 of which are local but on different devices, and 1 of which is offsite. This gives you tiers of recoverability.

It’s important from a backup perspective that it’s point in time as well, otherwise as soon as you get ransomware that encrypts your file you now have replicated those changes everywhere.

[1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

Re:Canon's cloud platform has lost users' files and can't re... by mumblemumble on Tuesday August 18, 2020 @09:29AM (#24198272)
I would add to that, that the offsite one should also be offline.
Re:Canon's cloud platform has lost users' files and can't re... by skoskie on Tuesday August 18, 2020 @10:39AM (#24199100)
That’s a lot harder to pull off. What methods do you use to accomplish this?
Re:Canon's cloud platform has lost users' files and can't re... by site-packages1 on Tuesday August 18, 2020 @11:27AM (#24199844)
Where I worked, weekly point in time backups were required. Those backups were put onto tape drives, those tape drives were set on a pallet and then driven by truck to an offline second location. IMO _that’s_ how it’s done properly.
Re:Canon's cloud platform has lost users' files and can't re... by zhengyi13 on Tuesday August 18, 2020 @12:30PM (#24200724)
When we used Iron Mountain at my last job, there were two interesting additional concerns:

1) That nobody in our company actually knew where they were physically stored (insider risk?), but

2) That we had assurance that the physical storage was far enough away that a physical disaster in our area wouldn't also touch where the offline storage was located.

There's a concomitant jump in RTO if that's what you do this, but hopefully that's well understood among the stakeholders.

Re:Canon's cloud platform has lost users' files and can't re... by site-packages1 on Tuesday August 18, 2020 @03:08PM (#24202917)
Definitely a commitment jump, and not always necessary. We ran our own data centers, off the topic of backups, and it was wild to me that they were so disaster proof. There were different power lines from different power companies coming in in case either of the power companies had an outage, and some giant diesel power generator that could last a long time while fully powering the data center.
Re:Canon's cloud platform has lost users' files and can't re... by mumblemumble on Tuesday August 18, 2020 @10:52AM (#24199313)
Well, once upon a time, when I worked at a smaller company, I kept a revolving collection of tapes at home.

A slightly larger company also used tapes, and stored them at an undisclosed (to me) offsite location.

A much larger company kept it all in datacenters, but the "offline" backups were disconnected from the WAN when they weren't actively being used.

Re:Canon's cloud platform has lost users' files and can't re... by reaperducer on Tuesday August 18, 2020 @11:14AM (#24199658)
That’s a lot harder to pull off. What methods do you use to accomplish this?

Depending on how much data you're backing up, sneakernet works.

When I was still in the office, my company had me rotate a set of backup hard drives between the office lockup and a strongbox in my house. The notion was that it was unlikely that both the office building and my house 12 miles away would both burn down at the same time.

Of course, now I'm working from home, so all of the eggs are in one basket again.

Re:Canon's cloud platform has lost users' files and can't re... by chiph on Tuesday August 18, 2020 @05:55PM (#24204811)
External USB drives for me. One set of drives is stored in a fire safe at home, one set of drives is stored at the office, and one set is stored at a relative's house in another state. The ones at home get refreshed most often, the ones stored at the office get refreshed about once a quarter, and the one at the relative's house get refreshed during holidays and family get-togethers.

The drives are encrypted (Truecrypt) since they will be outside my physical control. The ones at the office I am prepared to abandon should I get fired/laid-off.

Re:Canon's cloud platform has lost users' files and can't re... by randoramax on Tuesday August 18, 2020 @10:41AM (#24199138)
S3 bucket with object lock is as offline and convenient as it gets
Re:Canon's cloud platform has lost users' files and can't re... by wtracy on Tuesday August 18, 2020 @11:59AM (#24200279)
If you're going to pitch an AWS service as "offline", then pitch Glacier.
Re:Canon's cloud platform has lost users' files and can't re... by reaperducer on Tuesday August 18, 2020 @11:15AM (#24199671)
I don't know much about S3, but isn't S3, by definition, online, not offline?
Re:Canon's cloud platform has lost users' files and can't re... by csydas on Tuesday August 18, 2020 @05:45PM (#24204728)
AWS S3 (and a few compliant providers) offer immutable options, both in governance/compliance mode.

Allegedly, compliance mode is unalterable by any account period, I guess the equivalent of the immutable attribute without an overriding account. I'm not immediately familiar with any literature on attacks on this feature, but I've also not searched hard; however, I know from my clients that it's an accepted form of WORM, and that cloud storages like S3 are considered in the same vein as tape when immutability is in play.

I suppose it will be a case in the future that proves the efficacy of AWS/Azure/s3 providers, but for now, a lot of regulatory policies for 3-2-1 allow for such storage to fulfill the "2" part of the 3-2-1 rule.

Re:Canon's cloud platform has lost users' files and can't re... by zxcvbn4038 on Tuesday August 18, 2020 @11:48AM (#24200105)
I’ve always wondered if Amazon backs up S3. I don’t think they explicitly say but I get the impression that it is the user’s responsibility to replicate to a second region to guard against data loss so I am guessing not. Object Lock wouldn’t protect against an S3 failure.
Re:Canon's cloud platform has lost users' files and can't re... by count on Tuesday August 18, 2020 @04:05PM (#24203695)
They say it's designed to provide significant durability : https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurabili... (99.999999999% durable over a year per object, and able to sustain data loss in 2 facilities).

Given the peeks that amazon has provided into the scale of S3, I don't know if you CAN 'back it up'.

Re:Canon's cloud platform has lost users' files and can't re... by danielheath on Wednesday August 19, 2020 @05:40AM (#24208689)
As far as I can tell it’s replicated (by default - see “reduced redundancy storage”), not backed up.
Re:Canon's cloud platform has lost users' files and can't re... by ashtonkem on Tuesday August 18, 2020 @09:52AM (#24198526)
Or at least in an append only data store.
Re:Canon's cloud platform has lost users' files and can't re... by mumblemumble on Tuesday August 18, 2020 @10:38AM (#24199095)
Though, tread carefully there. If the thing that makes it append only is that you're just not using its destructive update features, then the store isn't really append only. Just because everyone agreed to limit the ways they interact with the data store doesn't mean you can trust buggy code, sloppy programming, and attackers to honor that agreement.
Re:Canon's cloud platform has lost users' files and can't re... by dec0dedab0de on Tuesday August 18, 2020 @12:34PM (#24200777)
I see a bunch of explanations, but I don't think any of them really drove home the reason for why you usually need both.

Imagine if you had a document that stored useful information. You had this document automatically replicated to another system in a different time zone every time there was a change.

You think you're doing great, if there is an outage in the one system you just get your important document from the other system.

Then one day someone accidentally copies and pastes the wrong data into the file. Now what do you do? If you goto your replicated copy it also has the bad data.

The answer is you should have had backups too. so you could go back an hour,a day or a week or even much longer. Depending on how much data you have, how often it changes, how important it is, and how quickly someone would notice bad data.

Re:Canon's cloud platform has lost users' files and can't re... by kgermino on Tuesday August 18, 2020 @09:24AM (#24198239)
I think the biggest factor is timing (although backups should also be on a second system).

In short, if anything that happens to the files is immediately copied to the "backup" then you don't actually have a chance to recover from any software problems. Whereas, if you make a copy of the data every night and keep the last 30 copies you can find an issue like this and go back in time to retrieve the files from before it started.

Re:Canon's cloud platform has lost users' files and can't re... by throwaway9d0291 on Tuesday August 18, 2020 @09:27AM (#24198258)
Replication is intended to create a 1:1 copy of the data. If data is removed from the primary system, replication will ensure it's removed from the replica.

A backup can be a 1:1 copy but it should be set up such that something going wrong with the primary (e.g. cryptolocker malware), can't affect it (since "something went wrong with the primary" is what the backup is intended to resolve).

To do that you could take the backup offline or use features like filesystem snapshots to ensure that changes can be rolled back.

Re:Canon's cloud platform has lost users' files and can't re... by jrochkind1 on Tuesday August 18, 2020 @10:13AM (#24198768)
The problem is that most replication setups will replicate deletes (and other modifications) as well. So if misbehaving software deletes (or corrupts) things, those deletes (or corruptions) will be replicated, and the replication does not give you a way to actually recover from a mistaken delete.
Re:Canon's cloud platform has lost users' files and can't re... by ResidentSleeper on Tuesday August 18, 2020 @09:25AM (#24198243)
Backups protect against a wider class of problems. For example, a software vulnerability/bug or a human error could result in the deletion of an object in all replicas (because deletions are replicated too), but you'd usually need another incident to occur for you to lose the backup as well.
Re:Canon's cloud platform has lost users' files and can't re... by SAI_Peregrinus on Tuesday August 18, 2020 @09:37AM (#24198361)
If your drive fails (or a single computer fails) replication saves your data, and keeps things running seamlessly.

If you `sudo rm -rf / --no-preserve-root` your drive, replication deletes everything, while backups let you restore to the last time you took a backup.

Re:Canon's cloud platform has lost users' files and can't re... by njharman on Tuesday August 18, 2020 @11:02AM (#24199466)
Replication includes replicating deletes (and overwrites, and subtilier changes). Replication is READ / WRITE.

Backups are WRITE ONCE. They persist if the original is deleted or modified.

Separate infrastructure and geographical distribution are orthogonal.

Re:Canon's cloud platform has lost users' files and can't re... by geerlingguy on Tuesday August 18, 2020 @09:23AM (#24198226)
Classically, backups are taken then stored offline (e.g. on archive tape, on optical media, or on drives that are disconnected and put in storage between backups). Otherwise, if they are not 'cold' backups, they are still disconnected so if the main storage blows up, it won't impact the data in the backups.

Replication usually involves storage that is powered up and connected somehow to the same systems as the main storage; and any data that's corrupted on the main storage would propagate to the replicated storage.

Re:Canon's cloud platform has lost users' files and can't re... by gameswithgo on Tuesday August 18, 2020 @09:48AM (#24198485)
A backup would be resistant to someone accidentally deleting all the files, for instance. Replication would not.