> Please tell me you verify your backups now and then?

Then one can't call it "set and forget", right?

Backup testing can be automated. I don't do this for my personal stuff, but at work there's a box that does a restore of our primary DB from backups, loads it into MySQL, and runs some smoke tests to make sure it looks roughly right. A quick script and a cronjob, and backups get tested every night.

I'm sure there's more thorough ways to do this kind of testing, but whatever level of confirmation you need automating it should be viable and then you only have to pay attention if/when something breaks.

Borgmatic runs consistency checks [1] once a month on all repositories and archives and I occasionally retrieve older versions for selected files (archives with --verify-data only once a year or whenever I feel the need - there's 9TB of data in the borg repo, which takes a bit to scan). Note though that borg is not my main backup, it is the fallback "3" in the 3-2-1 principle, where my primary data is a ZFS Raidz2 and my primary backup is an offsite ZFS Raidz2 in pull mode. I added borg because I did not want to rely on a single software (ZFS), although this fear was unstained so far.

[1]: https://borgbackup.readthedocs.io/en/stable/usage/check.html

You may also be interested in borgmatic's (beta) "spot" check, which compares your backup archive's files to your source files: https://torsion.org/borgmatic/docs/how-to/deal-with-very-lar...

Nice! This looks really awesome and practical. Thank you so much.

Sure thing! If you have any feedback on how it works (or doesn't) for you, please feel free to file a ticket.

This always gets repeated, sounds good and makes sense theoretically but in reality there's no good way to do that and it should be the job of a computer to that.

Restoring one file from the backup, works but what if something else is corrupted?

Restoring the system from the image, works but what if some directory is not in the backup and you don't see that while testing?

You are so wrong it hurts to read this shit on my screen. I've personally witnessed DOZENS of instances where a backup, thought of as "good", turned out to be un-restorable in the timeframe needed: total fucking paperweight of useless shit.

The point is not only to verify the data is good/not-good. The point is also to practice the procedure and build muscle memory.

If you don't practice important stuff, you're a dumb motherfucker.

borgmatic's "spot" check (probabilistically) protects against both of those failure modes: https://torsion.org/borgmatic/docs/how-to/deal-with-very-lar...

Indeed, I think these kind of automated checks are much more helpful than telling people they have to "test" test their backup. If a backup software doesn't do that automatically and reports if there's something off it's not good software or user experience.

I think the point is that if your data is valuable enough for you, you can't really trust that option in the backup tool to work - maybe you misunderstood some config option and the test now isn't really run, the tool is broken, or is run only on some of the backup files or dirs etc... or your original config might have missed a folder because it was mounted through some other filesystem (happened to me with Borg actually, and my whole /home/user dir especially wasn't backed up for the first 6 months I ran it :).

Seems to be good to have another tool that you either manually or automatically can setup to run regularly that tries to locate random files from your existing file system in the backups? Something like that.. though that other tool might be broken as well of course... :/

It's a very hard problem. In the end everyone is having an increasing amount of data where double checking it manually is not feasible any more and a perfect solution is maybe not possible.

Reling on software with good defaults that a lot of people use is probably a relatively safe bet combined with a second or third backup system (Personally I use Backblaze and Time Machine).