News for hackers. Stuff that matters.
Re:Correct Backups Require Filesystem Snapshots by throw7 on Monday May 16, 2022 @04:18PM (#31402688)
Wouldn't you need to quiesce the db or application before the snapshot?
Re:Correct Backups Require Filesystem Snapshots by jjnoakes on Monday May 16, 2022 @04:48PM (#31402969)
Not usually. A well-written application that cares about your data should be written in a way to survive sudden power loss, and to such an application, a file-system-level snapshot taken at an arbitrary point in time looks basically the same as sudden power loss.

Now, that being said, if you have the ability to set up backups where you can minimize the file system activity, you might be slightly better off doing it, but the ROI is probably fairly low unless it's extremely trivial to set that up for everything that's running.

Re:Correct Backups Require Filesystem Snapshots by compsciphd on Monday May 16, 2022 @04:51PM (#31403000)
that's not really true. there's a reason VSS exists on windows.
Re:Correct Backups Require Filesystem Snapshots by jjnoakes on Monday May 16, 2022 @05:01PM (#31403085)
Would you mind elaborating? I'm not a Windows expert so I don't know off the top of my head what VSS provides that helps poorly written applications survive sudden power failure more robustly than without VSS in the picture, and how that's better than a file system point-in-time copy-on-write snapshot.
Re:Correct Backups Require Filesystem Snapshots by compsciphd on Monday May 16, 2022 @05:30PM (#31403367)
databases might need to do lots of writes. (example, could be modify multiple rows in a db). They can't neccessarily capture it into a single write operation. they will issue multiple.

VSS is a method windows has to allow application to both be informed that a snapshot operation is in progress (and should quiesce their writes) as well as the ability to delay the snapshot operation from occurring (because they are in the middle of a "transaction"). If they delay too much, the snapshot operation will fail and reported back as such to the caller.

edit: also see the former maintainer of VSS at MSFT who responded to me in my other post in this thread

Re:Correct Backups Require Filesystem Snapshots by kaba0 on Tuesday May 17, 2022 @06:50AM (#31408556)
While the former maintainers post was very interesting, as I see it mostly helps with applications that already manage the sudden power loss use case. So a database can make use of that explicit API to somewhat better handle a snapshot, with no lingering transactions.

With only a COW filesystem snapshot, the end result will still be correct in case of programs that use transactions (as mentioned, dbs will either have commited all the transactions and the data is saved, or one was in-flight, those will be reverted and you are still in a consistent state). For programs that don’t support transactions both might be faulty, for those that support transactions COW will work correctly and for those that support this API COW will again work and might save a bit fresher version.

Re:Correct Backups Require Filesystem Snapshots by jjnoakes on Monday May 16, 2022 @05:57PM (#31403628)
Doesn't sound like that protects against power failure.

Most databases I use keep a write-ahead log, which guards against power failure without requiring an external service. Multiple writes are handled without an issue. If power is lost, the database recovers automatically using whatever is on disk, no matter when it was interrupted.

So a file system snapshot behaves the same and the database can start up from the snapshot even if it was taken at an arbitrary point mid-write.

Re:Correct Backups Require Filesystem Snapshots by tatersolid on Tuesday May 17, 2022 @07:44AM (#31409028)
In many if not most production databases, that write-ahead log will be on a different volume or file system from the database files. This is often recommended in documentation for performance reasons.

So now it is quite possible you back up an older WAL coupled with a newer DB file that has partial writes inside. Boom.

This “multi-volume, multi-writer coordination” is the sort of problem VSS solves on windows. I’m not sure there’s anything equivalent on Linux or MacOS that can do multi-volume snapshots consistently.

Re:Correct Backups Require Filesystem Snapshots by jjnoakes on Tuesday May 17, 2022 @04:47PM (#31416102)
In that case you just back up the database first and the WAL second.
Re:Correct Backups Require Filesystem Snapshots by blibble on Monday May 16, 2022 @04:34PM (#31402832)
yep, otherwise you risk some state being in memory and not on the disk
Re:Correct Backups Require Filesystem Snapshots by topspin on Monday May 16, 2022 @06:41PM (#31403973)
Your answer is too definitive. Applications and databases can indeed suffer power loss while losing nothing of value. An ACID database used by a correctly designed application will lose nothing on power loss; state "in memory" is uncommitted in such a system effectively doesn't exist. It never happened and, furthermore, no one cares.

The term of art is "crash consistent," and any ACID database must preserve all committed state across events such as power loss. Such a database is correctly backed up when copying a simultaneous point in time snapshot across all involved volumes.

Not all databases are truly ACID. Lots of software relies on uncommitted database state. But we're talking about a solved problem here; if you require ACID behavior the means to achieve exactly that are available. Any exception to that statement, including hardware misfeatures or lack of two phase commit across databases, is equivalent to "incorrectly designed."

In a correctly designed system quiescing the database isn't necessary, but might still be used as a precaution or a performance optimization.

Re:Correct Backups Require Filesystem Snapshots by blibble on Tuesday May 17, 2022 @02:18PM (#31414489)
this is fantastic and all for carefully written databases (none of which have ever had any bugs)

unfortuntely most applications aren't like that and assume the filesystem is both atomic and durable

Re:Correct Backups Require Filesystem Snapshots by topspin on Tuesday May 17, 2022 @02:34PM (#31414697)
> 99%

Your figure is too high. Commercial relational databases, many open source databases and widely used file systems are all sufficient. Oracle, SQL Server, DB/2, PostgreSQL and MySql/MariaDB (with InnoDB) are all ACID systems and recover from crash consistent backups fine. Ext4, XFS, NTFS, etc. are all sufficient file systems, unusually in default form. Reliable storage devices are widely available.

Many applications designed around these platforms use COMMIT properly. When they're don't quiescing the database probably won't help with backups anyhow; all you're doing is deferring writes and if the application is not ACID then you can still lose. The database itself, however, will recover and since we're tossing probabilities around, that's 99% of the battle.

Re:Correct Backups Require Filesystem Snapshots by blibble on Tuesday May 17, 2022 @02:54PM (#31414928)
I'd bet quite a large amount of money that if you could see every process running on every server on earth: less than 1% of the total would be database processes

meanwhile most of the rest seek randomly in files, buffer and write data without calling ever fsync, or append without a pointer and appropriate write barriers (I'm looking at you journald)

Re:Correct Backups Require Filesystem Snapshots by topspin on Tuesday May 17, 2022 @09:47PM (#31418386)
I don't know what your basis of measurement is, so I'll continue to ignore your ratios. Tolerance of crash consistent state isn't exclusive to databases. Filesystems themselves are often designed to tolerate, for example, sudden power loss without corruption. Applications that manage large collections of files can also achieve this; maildir is an example of such a scheme. All of these systems are using related principles of consistency, and all of them can be correctly backed up by copying volume snapshots.
Re:Correct Backups Require Filesystem Snapshots by blibble on Wednesday May 18, 2022 @06:25AM (#31421051)
are you ignoring the main point too, or simply missing it entirely (repeatedly)?

the fact there exist some applications (databases, applications using maildir) that correctly use write barriers doesn't mean all applications are always consistent on disk

Re:Correct Backups Require Filesystem Snapshots by SoftTalker on Monday May 16, 2022 @04:53PM (#31403015)
Exactly. A snapshot is a point in time on the storage media. It allows you to make a backup as if the backup occurred at that point in time, i.e. files are not changing while the backup is running. A snapshot is not a guarantee that any particular file is consistent with the state of the application that is using it. It's still a backup of an "open" file with all the potential issues that implies.