From someone smarter than me: people think they need backups while in fact they need restores…

That’s one of my preso jokes. “Backups are a worthless waste of time! Restores, on the other hand…”

The canonical link:

https://www.joelonsoftware.com/2009/12/14/lets-stop-talking-...

In the late 1990s, W. Curtis Preston wrote a book called “Unix Backup and Recovery”. A later version was just named “Backup and Recovery”. Both were published by O’Reilly.

He was saying similar things at that time (e.g., “Backups are worthless, Restores are priceless”), and I’m certain he wasn’t the first.

But you’d have to ask him who was saying things like that before he did.

Disclaimer: Curtis was a co-worker of mine at the time, and I was a technical reviewer of the book in question.

Beautifully succinct.

The article slightly annoyed me by saying, “Why one does backups? To have a copy of data at hand if something goes wrong.” Not, I think, the best way to put it, per your comment.

How do you go about testing your backups? Like if you have 1TB of data, testing that all of it is valid is not trivial, and I guess that should be done every time you create a new backup set? Shouldn't the backup tool somehow perform this test automatically?

> testing that all of it is valid is not trivial

If it was easy then people wouldn't have to be reminded to do it—it'd just get done. :)

One way could be to pre-compute checksums of the files in question and then verify them periodically:

* https://aide.github.io

* https://packages.debian.org/search?keywords=aide

It's also why people suggest using ZFS: it has checksums on the live data to make sure bits aren't flipped, but those checksums can then be replicated to remote copies (zfs send/recv) to make sure your copies also coherent. A service like rsync.net allows remote ZFS data sets—that can also be encrypted.

Being 100% confident you've got a bit for bit copy of what you think you have, doesn't mean you have a restore process that works.

Unless you've tested and proven to your self you can bring up a working system from your backups, you've only done the first half of your disaster recover work.

For me, that means restoring onto your cold spare hardware (either identical or similar-enough for you), shutting down your system pulling it's drives and replacing them with blank ones and restoring onto that (but be careful you can be sure you don't need that specific hardware/firmware/peripherals - cause in 18 months time when someone backs a truck into your office and load all your electronics into it you might not be able to get an identical system), or restoring a running system onto newly provisioned instances on cloud-provider-of-choice.

Prove to yourself at least, that you can have business continuity in a know amount of time once you pull the pin on your DR/restore-from-backups plan.

Yup, this is it. And you don't need to do it for every single backup. Before running the backup, do four coin flips. If they all end up heads (adjust the number of flips based on or how much time you can afford to spend on backup testing) you test the full recovery process of the impending backup.

Of course, it will depend on several parameters, but in my experience, doing a thorough test for a random subset of items is often more economical than a half-assed test of all items.

As a bonus, the random sampling will let you infer things about the totality of all items. (As opposed to any other scheme for selecting which items to test.) So once you've run 27 tests and only one failed, you know at least 85 % of your backups work. At 1/20th of the cost of testing them all, this is a good deal on information.

There is no alternative to performing an actual restore periodically in order to verify that backups and the restore procedure do work.

>should be done every time you create a new backup set

That could be prohibitively expensive. For personal use, each time I buy a new computer (desktop / notebook alternating) I use it as an opportunity to test my restore procedures. I restore my most recent backup to the new computer, then verify that it operates the same, and has all the same data stores, as my next newest computer (the one whose backup I restored).

Not just test the backup, but practice and ensure you are familar with the recovery process as well. Having a clear plan of what you will do if that disk fails/machine is lost/etc. is better than troubleshooting and fumbling with tools while the clock is ticking. A written procedure may be helpful as well (I have one for my Proxmox server for setting up the machine and restoring the VMs from the backup disk).