Canon's cloud platform has lost users' files and can't restore them

Re:Canon's cloud platform has lost users' files and can't re... by rsync on Tuesday August 18, 2020 @02:02PM (#24201980)

"For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment?"

I can only describe what we do, and, of course, there is an enormous scale difference here, but ...

Every single rsync.net account is it's own ZFS filesystem which means every single account gets snapshotted[1] nightly on a schedule. This means that the enormous operation of "backing up" all of rsync.net happens in small, manageable, chunks.

Of course, the ZFS snapshots of a customer account are not "backups" per se, but if a customer choose "geo-redundant" storage in another facility, it is those very same snapshots that we (zfs) send over. Those are, indeed, backups.[2]

The most interesting part, in my opinion, is that the daily/weekly/monthly snapshots are immutable. So you can publish your rsync.net credentials or suffer a disgruntled employee or ransomware attack, etc., and those snapshots remain safe - they are read-only.

[1] Accessible, browseable, in ~/.zfs/snapshot

[2] GR storage costs 1.75x normal pricing.

[ Parent | on HN ]

Re:Canon's cloud platform has lost users' files and can't re... by RantyDave on Thursday August 20, 2020 @06:32PM (#24229801)

So it does not assert that the filesystem is in a consistent state before it snapshots. IE we could be midway through applying a database transaction; or there could be a reference in one file to another that doesn't exist (yet).

It could be argued that one needs to do a site wide "write cache flush", stop, and snapshot. Not that I think for one second the service provider should (or could) be in a position to detect when a good time to snapshot might be....

[ Parent | on HN ]