I just wish that storing things like keys on paper was easier.
At CoreOS we put some keys on printed QR codes and scanned them with an airgapped laptop every 90 days to confirm the keys were safe.
I agree. But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business.
And then there's the sheer stress of being responsible for the backups, but not being able to test bare-metal recovery.
I'm interested in backup ("what kind of weirdo is this!"), but that wasn't a fun responsibility.
Sometime your job is not to be able to take the backups, but to take the blame when a restore can't be done. If you think that might be you, time to polish up that resume...
Especially for cloud-based businesses, refusing to let you spend ~8 hours of production platform costs to run a full platform rebuild fire drill is insane, that's about 0.1% of your annual prod AWS budget, and 0.5% of you (or your team's) annual time.
The boss owned the company; I think it was shortsighted of him. Being able to blame me wouldn't have done him much good, once his firm had gone down the tubes, and I'd moved on. But it was his business to lose, and he knew what his margins were - I didn't.
He didn't like me much, mainly because he was a control freak, and I knew his systems much better than he did - I had the control that he craved. Control freaks shouldn't hire people that know more than they do.
There are very clear directions to my friend for when it is acceptable to access my stuff, and very clear consequences for malice.
I have two friends who've given me private key fragments (from a Shamirs Secret Sharing setup) with the explanation "If anything happens to me, you'll work out who you need to talk to and what you'll need to do."
I haven't done that myself, because I don't have any need/desire for anybody to decrypt my backups once I'm not around. That might change if I end up with dependants one day.
Fire drills are important, but I get a Telegram notification every day when my rsync jobs complete. The notification is muted, but I see it in the list every morning when I use Telegram to text my girlfriend. If it wasn't there, I'd immediately know something wasn't working.
A quick backup verification is not enough - I learnt this the hard way and lost 9 months of data only after I did full annual DR test. The backups were setup and configured by the (very well known) manufacturer of the backup software. They screwed up on the DNS name
Why not just remember the password and perform a regular fire drill decryption to ensure you won't ever forget it?
I have been involved or aware of a few nightmares where an expected loss resulted in all kinds of pain and problems for the family has they lost access to critical accounts, financial data, and other things that required lengthy interactions with government agencies, banks, etc that would not have been needed if the data would have been able to be recovered from the systems
I use a crazy long passphrase to encrypt my backups, but should I forget - it is also printed on archival paper inside a sealed envelope in a friends safe deposit box (I also have a copy of his backup passphrase for mutually assured destruction :)).
Also, every once in a while run a fire drill and actually restore something from each of your backups. This is when you find out the rsync job has been stuck for the last 80 days.