Snapshots are stupid in the case of a content addressable, immutable data store.

You're better off asserting that your objects haven't changed (Which they weren't, and I agree that they should have been) and were valid in the first place (See above).

With snapshots, you'd invevitably want to dedupe them, which would be basically the same thing since it's append only, but with the dedupe infrastructure as another failure point.

If you're using copy-on-write snapshots, then the total size of your snapshots should be small since most of the data in said immutable content store never changes. But the benefit is a bit-error between one mirroring operation and the next doesn't overwrite your unchanged, good data on the slaves.

The problem I think needs more attention here is ext4 silently corrupting data. ZFS has it exactly right with the built-in checksumming on write and read - it can't stop a disk going bad, but it can tell you exactly what's affected and _when_ - corruption would've been caught the moment the mirroring operation tried to read back bad data (and would've faulted the process, rather then happily return bad data).

The point of a backup is to have redundancy, if backups are too integrated with their target, it becomes one complicated system together (that needs backup). This holds as a general rule.

> Snapshots are stupid in the case of a content addressable, immutable data store.

Well, the KDE incident proves otherwise.

Well, not necessarily. The issue is that filesystem corruption lead to undetected Git repository corruption, which is what made it possible to push corrupted repos to the mirrors.

It would have been just as easy to push those corrupted repos to all of the backup tapes in the rotating snapshot set. A snapshotting filesystem could be a good backup (and seems to be what one of the sysadmins is pushing for).

But even more important is to fail fast and identify git repo corruption as soon as it can be detected so that further damage can be avoided.

Git is not an immutable data store. The refs are very mutable and change-sets get garbage collected.

The KDE sysadmins are well aware of that, at least. Mutable operations that would leave dangling blobs cause a backup copy of the appropriate ref to be generated before the force-push/branch-deletion/etc. are run so that there's nothing for git to garbage collect.

Yeah, if you're incapable of accepting that complicated scenario is complicated.

The next two paragraphs identified two things that they weren't doing that they should have been. Otherwise they'd just have lots of snapshots of bad data.

For everything with a sha1 hash, I see where you're coming from. And most of the data in a repo is covered by them. But things like tags, branches and reflogs don't themselves have hashes, they are just metadata referencing content in the append-only store. It sounds like they were backing up their reflogs, which is great, so they could recover if a user, say, accidentally deleted all the branches off the central server.