Whereas non-snapshot backups don't have that property; the material in the backup is not necessarily identical to any past state of the filesystem that existed. It's something like a past state plus random roll-backs of files, or portions of files, from multiple previous states.
Guess which of these situations applications are much more likely to be able to recover from (if any at all)?
When people write crash recovery code, they typically assume that the world simply stopped, not that it stopped, and then some data was randomly rewound to unspecified older states.
I don't think that you can reasonably defend against data that is restored from a backup, where the oldest part of the backup is an hour older than the newest.
A backup is like a raster scan image of a fast moving object. What should be a rectangular train car looks like a parallelogram: it's not a picture of any scene that existed. Imagine that the raster lines are randomly sampled (not a progressive scan, or even interlaced) and now recover a sane image.
If your application relies on multi-file collaborated persisted states (for example, an append-only log and a database snapshot), application can make reasonable assumptions on when a file state is committed, such as a database snapshot is `fsync`ed before the append-only log started. Ideally, even on a filesystem without transaction support, this order is preserved in time.
However, it may not be preserved for a "file-to-file" backup system because it can loop over the append-only log file first before the database snapshot, causing ordering issues. That could result a "ABA" problem, where the append-only log is corresponding to an older database snapshot, and potentially causes issues.
That has been said, is this a common scenario (multi-file persisted state) for applications? (I believe SQLite handles this particular ordering issue fine). I am not sure, and just want to call out filesystem snapshot solves a very particular problem.
If your application relies only one file for the state, or these files are orthogonal to each other (.xlsx or .docx or any .markdown, .mp4, .jpg files), it is a non-issue. And if you need to backup a database, you better do that with the database provided tools.
The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happened.
If the application structures its scattered write transactions such that they write in increasing offset order, then maybe it's okay.
As far as I understand, the only thing snapshots make sure of is that you're looking at a consistent state of a filesystem from the point of view of the filesystem, so e.g. if there was a sequence of operations "remove file a/b", "create file c/d", then you're not going to record a state with both files "a/b" and "c/d" present --- something that could happen with a backup tool not using filesystem snapshots, if it reads directory "a" first and directory "c" second. But there are no assurances about application transactions. So if KeePassXC has a bug, then if there is a sequence of operations "remove password A", "add password B", then a backup tool may pick up a version of KeePassXC database which is going to contain both passwords A and B, even if the backup tool uses filesystem snapshots.
[1]: At first I didn't get the example from TFA about Firefox and SQLite, but I've reconsidered and it makes sense now. I'll still say that you may get an intermediate state in a snapshot. However, snapshot prevents some classes of inconsistencies (though not all). Call me convinced about ZFS.
The serial copy of the file does not have that property. We may be able to find some k such that the backup contains some writes newer than k, while missing some older ones.
The snapshot situation looks, to the application, like a crash and reboot, which it knows how to deal with (with test coverage and everything). The application recognizes that there is an uncommitted transaction and rolls things to the prior state. Or else sees that there is enough info to finish the transaction; just the commit part wasn't done.
The missing old writes situation doesn't seem easily recoverable; I have no idea how you would code defensively against that, other than assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing.
Actually, that's the opposite of what you want: if the backup and application are racing in the same direction, then every write has a coinflip opportunity to either be included in the backup (if the application is slightly ahead and writes that block just before the backup reads it), or be ommited from the backup (if the backup is slightly ahead and reads that block just before the application writes it).
Whereas if the application writes from the end of the file to the start, the write where it meets the backup coming the opposite direction will be arbitrarily kept or lost, but in theory (but not in practice), any writes before it (toward the end of the file) will always be kept, and any writes after it (toward the start of the file) will always be lost, which is the same semantics you'd get from a snapshot at the time of that write.
The point of the article is that, after a restore from a backup that used a snapshot, whatever ends up on disk, it’s something that the application’s developers could have planned for.
On the other hand, if the backup didn’t use a snapshot, the application may be given something that those writing the application couldn’t have foreseen.
I guess not many applications will handle either case perfectly or even decently (who even tests for the related issue of disk full errors on save nowadays?), but databases should.
I also would expect/hope lots of embedded software with a file system to handle this well, not because they get restored from faulty backups much, but because they have to be able to recover from unexpected reboots.
The idea is that you'll only encounter corruptions that the application could already hit due to crashes or power outages (and hence hopefully supports recovering from). For example, with naive reads, you might:
read the first half of the file
context switch to the application
application writes to the first half of the file
application fsyncs previous writes
application writes to the second half of the file
context switch back to you
read the second half of the file
and end up with data in the second half of the file that the application normally only writes after it's sure that corresponding data has been written to the first half.
A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption).
Thus, any discussion on file corruptions without a file system that supports transaction, requires every application to use a competent database underneath (SQLite) at least.
And even with transactional support in a file system or using a database, you need every application to have the correct business logic that does the transaction correctly as well.
All-in-all, correct backup cannot be solved universally without knowing all the applications. The best we can do is to probabilistically avoid obvious issues, i.e. using FS-level snapshot.