And the most important part of the rule which they forgot: Test your backups.

I once has to restore a DB from backups just to discover that the backups from 2 days ago were corrupted. Fortunately backup t-3 days worked. And I had to do some binlog mongering to recover the rest of the data.

Test your backups people. They WILL fail.

From someone smarter than me: people think they need backups while in fact they need restores…

That’s one of my preso jokes. “Backups are a worthless waste of time! Restores, on the other hand…”

The canonical link:

https://www.joelonsoftware.com/2009/12/14/lets-stop-talking-...

In the late 1990s, W. Curtis Preston wrote a book called “Unix Backup and Recovery”. A later version was just named “Backup and Recovery”. Both were published by O’Reilly.

He was saying similar things at that time (e.g., “Backups are worthless, Restores are priceless”), and I’m certain he wasn’t the first.

But you’d have to ask him who was saying things like that before he did.

Disclaimer: Curtis was a co-worker of mine at the time, and I was a technical reviewer of the book in question.

Beautifully succinct.

The article slightly annoyed me by saying, “Why one does backups? To have a copy of data at hand if something goes wrong.” Not, I think, the best way to put it, per your comment.

How do you go about testing your backups? Like if you have 1TB of data, testing that all of it is valid is not trivial, and I guess that should be done every time you create a new backup set? Shouldn't the backup tool somehow perform this test automatically?

> testing that all of it is valid is not trivial

If it was easy then people wouldn't have to be reminded to do it—it'd just get done. :)

One way could be to pre-compute checksums of the files in question and then verify them periodically:

* https://aide.github.io

* https://packages.debian.org/search?keywords=aide

It's also why people suggest using ZFS: it has checksums on the live data to make sure bits aren't flipped, but those checksums can then be replicated to remote copies (zfs send/recv) to make sure your copies also coherent. A service like rsync.net allows remote ZFS data sets—that can also be encrypted.

Being 100% confident you've got a bit for bit copy of what you think you have, doesn't mean you have a restore process that works.

Unless you've tested and proven to your self you can bring up a working system from your backups, you've only done the first half of your disaster recover work.

For me, that means restoring onto your cold spare hardware (either identical or similar-enough for you), shutting down your system pulling it's drives and replacing them with blank ones and restoring onto that (but be careful you can be sure you don't need that specific hardware/firmware/peripherals - cause in 18 months time when someone backs a truck into your office and load all your electronics into it you might not be able to get an identical system), or restoring a running system onto newly provisioned instances on cloud-provider-of-choice.

Prove to yourself at least, that you can have business continuity in a know amount of time once you pull the pin on your DR/restore-from-backups plan.

Yup, this is it. And you don't need to do it for every single backup. Before running the backup, do four coin flips. If they all end up heads (adjust the number of flips based on or how much time you can afford to spend on backup testing) you test the full recovery process of the impending backup.

Of course, it will depend on several parameters, but in my experience, doing a thorough test for a random subset of items is often more economical than a half-assed test of all items.

As a bonus, the random sampling will let you infer things about the totality of all items. (As opposed to any other scheme for selecting which items to test.) So once you've run 27 tests and only one failed, you know at least 85 % of your backups work. At 1/20th of the cost of testing them all, this is a good deal on information.

There is no alternative to performing an actual restore periodically in order to verify that backups and the restore procedure do work.

>should be done every time you create a new backup set

That could be prohibitively expensive. For personal use, each time I buy a new computer (desktop / notebook alternating) I use it as an opportunity to test my restore procedures. I restore my most recent backup to the new computer, then verify that it operates the same, and has all the same data stores, as my next newest computer (the one whose backup I restored).

Not just test the backup, but practice and ensure you are familar with the recovery process as well. Having a clear plan of what you will do if that disk fails/machine is lost/etc. is better than troubleshooting and fumbling with tools while the clock is ticking. A written procedure may be helpful as well (I have one for my Proxmox server for setting up the machine and restoring the VMs from the backup disk).

People often confuse the Different Media, and this blog post makes it murky as well

>> The thing is, while keeping data on the same storage media, you may lose them due to the same hardware issues. In other words, you may lose two copies in the same accident. That’s why you should always combine media.

For "same media" the key part of this is storing the data on two DIFFERENT DEVICES of different types, they both could be hard drives, but they need to be in isolated systems of different types (say a Windows System and a Synology NAS, or a FreeNAS Storage Appliance and a Linux Server, etc)

The rule came about because people would use the same tape library, or have multiple copies on the same SAN (often times they would have a SAN Cluster of 2 or more devices that act as 1, and because they had 2 "devices" they felt they were protected.

Disparate media type (HDD, Tape, DVD, etc) may have some advantaged but as long as you are putting 2 copies on say your Home Desktop, and a NAS you have satisfied 2 media's even if both are using Hard drives

That said today the most common way for a person fill offsite and separate media is to use a Cloud Backup of some kind.

> the most common way for a person fill offsite and separate media is to use a Cloud Backup of some kind.

Which has a different failure mode to disks/NAS/fileservers.

AWS/Dropbox/Google/BackBlaze et al. are all at risk from credit card and/or financial failures. I can go broke and be unable to pay my bills, but my hard drives will still store my data.

Google and AWS (to a lesser extent) are also occasionally subject to arbitrary and capricious account revocation. Google specifically worries me there, there are way too many stories of people have problems with, say, they Google Play developer account or their YouTube account, and finding they're cut off from their gmail and google drive access, with pretty much zero way if fixing it unless you have a million twitter followers or a Google insider to go to bat for you.

Sure, but there is where the 2nd copy comes into play in the 3-2-1.

3 Copies, 2 Media, 1 off site.

One should always have a local copy of their backups.

A couple decades ago, I used to keep backups of my desktop computer on DAT tapes. I had several different tape cassettes that I rotated through to keep many separate backups of all my data, and I thought of them as different devices until the day when I ejected one of the DAT tapes from the drive and it came out trailing a couple meters of unwound tape, with a small (but apparently important) spring hanging off the end.

Lesson learned! I now still had several intact tape backups of my data, but no functional device that could read and restore from them.

Oh oh, I forget to add that I would _never_ trust flash memory (eg. SSDs) to keep data for more than ~ a year without being powered on. It's absolutely a terrible archival format. (Powered on with scrubbing you'd at least know when it's starting to degrade).

> The different media types is probably the hardest issue; I'd wager that for most people, backing up > 2 TB on anything but spinning rust will be impractical/prohibitively expensive.

There's LTO (Linear Tape Open), LTO-8 has 12 TB of raw capacity and costs about 65 - 70 €. The newest variant (not yet _that_ widely available), LTO-9 has 18 TB.

You need a few to handle rotation, but for most that means at max 10 plus one new per year for the long term archive.

We provide Tape support in our open source Proxmox Backup Server product, it can handle single drives and drive robots (with auto exchangers that lessen the work on tape rotation), the data is deduplicated, compressed and optionally encrypted.

https://pbs.proxmox.com/docs/tape-backup.html

PBS can also efficiently mirror to remotes:

https://pbs.proxmox.com/docs/managing-remotes.html

Check the introduction/main feature section for more info if you're interested: https://pbs.proxmox.com/docs/introduction.html

The cost of the tapes isn't what keeps me from having it home. It's the cost of the tape drive.

I mean, it's a one time investment that's may be good for a decade or so, depending on the setup and workloads.

But yes, forking over a 2 to 4k for a new LTO-7+ changer isn't a small investment for private use, for a company it's IMO a no-brainer though.

FWIW: You can get them often cheaper, e.g., at ebay or whatever your local online reseller space is. For example, one can get LTO-5 ones here for 300 - 400 bucks and LTO-6 for 600 to 800. While LTO-6 only has 2.5 TB of space you can also create tape-sets (e.g., spanning multiple tapes) and for the more important data it may even be enough.

ages ago there also was always the recommendation that you should have two drives, so if one gets out of alignment you'll notice when it starts to create tapes that proper drives can't read anymore. Not sure if that's still a concern with modern ones?

> Not sure if that's still a concern with modern ones?

No, that's not really a concern currently, and I actually never heard that in combination with LTO - which is relatively modern.

Good to know. Quite possibly folklore that got transferred over to LTO.

>The different media types is probably the hardest issue

That's probably overrated and you shouldn't feel bad about not implementing it. The justification for it is:

>you may lose them due to the same hardware issues

buying different hard drive models/manufacturers/batches achieves the same thing without the risk of having data stored on dinosaur media.

I agree with that with one important caveat.

I wouldn't trust multi terabyte drives in raid 5 sets to not suffer cascading failures during a rebuild after a single drive failure/replacement.

I'd consider two file servers or NAS boxes with the same type/brand/model of drives in raid 5, to be "the same media type" in the context of that advice.

>A slight twist of this: I now have data old enough that accessing it with modern computers is starting to become a challenge.

I have that problem!

I have files in my Documents folder going back to the 90s and Classic macOS!

> I have files in my Documents folder going back to the 90s and Classic macOS!

Same. And while I can be sure I have them safely stored, I know for sure I have files I can no longer use because I don't have a way of running the applications needed to work with them. I must have _dozens_ gigabytes worth of copies of Zip disks and Syquest cartridges full of Macromedia Flash projects and Mac OS9 Filemaker Pro databases.

Why not set up an emulated virtual Mac? It's not very hard these days, and the performance is not a concern with modern devices.

My oldest files are from 1978.

My personal interpretation/variation of this is to use two different backup formats, even if everything ultimately ends up on hard drives (whether my own or a cloud service provider's). I usually will do 1 file backup via Duplicacy (but Restic/Borg/Duplicity/Time Machine work just as well) and 1 system image backup via Macrium Reflect/Carbon Copy Cloner. The file backup gets copied to Backblaze B2, the image backup stays local.

There are Blu-ray Discs specifically engineered for long-term archiving and have BER guarantees (anomalous bit read per gb of data stored per year archived or something).

Any more info on this like a brand name?

It used to be "M-DISC", with the 'M' meant to mean "millenium": as in, they'd last 1000 years.

I somehow thought the company making the M-DISCs was dead but apparently Verbatim (and others) still sells BluRay discs labelled as "M-DISC" and they're compatible with many BluRay readers/writers.

You can still buy M-DISC branded discs - either blu-ray or DVD. I think it matters more for DVD than for blu-ray because blu-ray discs use a more reliable recording medium anyway. People say there's little if any difference between M-DISC BDXL and 'normal' BDXL. You can get 100GB or 128GB BDXL discs.

There are two different Blu-ray dye technologies, LTH and HTL. HTL is the original and uses more expensive inorganic materials (germanium, bismuth, or palladium) and then there's the cheaper LTH that came later and uses an organic dye that's more susceptible to light, breaks down rather quickly over time, and has more burning/reading issues. You don't want to use the latter.

Typically if the surface of the disc is gold and darkens with writing, it's an LTH disc.

They include external USB drives and NAS as different types, specifically about the NAS:

> can be considered statistically independent too since it is connected over the network and may survive if something bad happens to a part of your infrastructure.

So my interpretation is: e.g. a HDD connected over SATA, an external HDD over USB and a HDD on NAS are all considered different media types even though ultimately they are all based on spinning hard drives.

Because of the different interfaces to said drives? (USB vs. Network vs. SATA)

yes that seems to be the point. NAS is over the network, USB is supposedly "cold storage" (unless you keep it connected 24/7). Imagine something like: all drives on your desktop PC are compromised due to a PSU malfunction that causes a current or voltage spike - in that case, both NAS and USB cold storage would be safe.

> The different media types is probably the hardest issue;

For my home stuff, I'm comfortable enough with having a backup on (raid 1 mirrored spinning rust) on my media server, plus a copy of that backup (also on spinning rust) on an external USB hard drive. The chances of simultaneous corruption of those - even though they're both spinning rust - seems "low enough" for me.

(I also have another raid 1 pair of spinning rust that powers up once a week very early Monday morning and rsyncs the media server backup directory, then powers back down when it's done, to mitigate against fat-finger-fuckups and/or home network p0wnage. Somebody who p0wns something inside my network might find the cronjob or shell scripts that run the wifi powerpoint those drives use on and off, but at least I'm down to an hour or so a week where that copy might get cryptolockered...)

> I'd wager that for most people, backing up > 2 TB on anything but spinning rust will be impractical/prohibitively expensive.

You'd lose that bet. 2 TB on LTO Ultrium tape costs under $10 (sometimes under $5 depending on volume of tapes ordered).

With the slight downside that you won't be able to acquire a drive for less than $800.

LTO's combination of cheap media and expensive drives is great for people with rooms full of tapes, but it makes it pretty unattractive for everything else.

Depends on your requirements. Something lowest common denominator like LTO2 is still useful, cheap, reliable and the drives are also cheap and reliable.

You can always feed your LTO2 tape into a LTOx drive and get the data back.

Obviously you have to be fairly patient, but for my use case (once a week on my third tier of backup) it's good and gives peace of mind that I can put in a fireproof safe.

Primary backups are still on the cloud though - S3 Glacier is a good option.

By LTOx, you mean at most LTO4 right? I thought LTO drives could read at most 2 generations prior. Also, you need to get to LTO5 to get useful capacity (~1.5GB/tape), LTO2 is limited to 200GB + whatever you can get with compression.

I've been keeping an eye on tapes for literally a decade or more, but for my personal backups hard drives get enough bigger every year that I'm comfortable with (for now) a pair of 8TB usb drives with secondary local backups which I swap between plugged in at home and powered down in my locked desk drawer at work every Monday. I've got a while to go yet before I'm worried about capacity there, and 12TB drives are already available, and will easily be in the right budget soft spot for me when I need them.

And how much is the tape drive? For _most_ people (not data centers) that prohibitively expensive. A 2 TB harddisk is $48 from newegg (and I'm sure you can find cheaper).

Lots of people here focusing on the cost of the drive without factoring in:

- For home use, it is likely a once-or-twice-in-a-lifetime purchase (try saying that about any other kind of media)

- You don't have to buy a brand new drive of the latest generation at sticker price; used last gen(s) gear from the enterprise works just as well.

- Perhaps most importantly, what's your data worth? I don't know about you guys, but I've got photos and documents that are irreplaceable.

Tapes beat all other storage media on $/GB, reliability over time, and arguably durability, all of which are the most important factors for offline backup.

$500 comes out in the wash over a decade or two. That's the kind of time scales we're talking about. Yes, it's not cheap and easy consumer electronics you can buy off the shelf at Walmart, but it's not unreasonable either.

Can you recommend specific tape drives and/or generations? I have never considered it but you make a good point. Maybe this is the time create a backup of my irreplaceable Google photos collection.

Really, it comes down to your budget. LTO drives don't do anything interestingly different across manufacturers, and I've seen no evidence of significant reliability differences either. So, just buy the biggest generation that makes sense for your data storage needs.

Two important things to note:

1. LTO5 and onwards have a feature called LTFS, which allows you to address the tape as an ordinary file system. Before that, you were limited to purpose-built tape tools.

2. Generally, tapes can be read in drives 2 generations newer, and written in drives 1 generation newer. This rule was broken with LTO8, due to new tape composition. Starting here, it's 1 generation newer read/write.

For "purpose built tape tools", tar and dd are fine. tar is a bit easier to restore. If you are running Windows, well, don't run Windows as Microsoft have removed all the built-in tape stuff they used to supply with Windows Server and all the alternatives either cost money or are hideously complicated.

Debian + old HPE server + tape drive + tar. Job done.

That's hilarious. I did buy a tape drive. Heck, I bought a tape library. Guess what? It's useless today because my data consumption has scaled with technology and that old DAT based solution just can't cut it. Any tape drive bought today will be garbage in N years. Just a simple fact.

With all due respect, your data usage needs are yours and can't extrapolate to everyone else's. A few 320GB DAT cartridges may be enough to back up the essentials.

How does one choose a tape drive? I see wild variation on tape drives and no idea on why that is.

If the devices are not too large or ridiculously expensive, than it definitely makes sense to use tape as backup. Hard disks are convenient but fail at inopportune times.

And the drives are expensive.... Especially if you want something that handles fairly modern tapes for actual useful capacity.

And how much for the tape machine?

The person above was talking about the cost of HDDs, they didn't include the cost of a machine or NAS to run them in so why should I?

Seems like a double standard wherein people are going to pretend housing multiple HDDs costs $0 (unrealistic) but won't even evaluate spending money on a tape drive.

You what doesn't get cryptolockered? Yesterday's tape sitting on the shelf.

Because we're talking about most people, and most people already have a machine into which they can plug a HDD. Often that's the very machine with the documents they want to backup.

I think "plugs into SATA or USB" is a reasonable standard for a backup storage medium.

Whereas I think that isn't a backup at all, because it is plugged into the same machine (i.e. online backup) and therefore going to get crypto-lockered.

A USB drive hooked up via ESATA or USB is directly comparable to an LTO tape.

So is a tape left in the drive.

If you can move a tape to a shelf, you can unplug a USB cable.

And to your question about having a separate machine, a Pi will be fine.

Neither SATA nor USB implies permanence.

Most people do not have a $1000 LTO tape drive.

You can pick up a second hand HPE deskside server with a tape drive included for $200-$300. Might only be LTO2 but that's still 400Gb of rock solid storage for no money.

If you cannot fit the backup on a single tape then you increased the headache and number of ways this can fail. I cannot phantom how you can see this as superior to hard-drive (which will probably end up cheaper too).