What surprises me is that you can extract the private key at all - I would expect it to be firmly inside a HSM, that only accepts signing requests and the key never leaves the module & the HSM wipes the key if it detects tampering (there is usually a battery inside to power the tampering detection even if the device is not plugged in).

So just exporting the private key so easily without some pretty involved hight-tech HSM key extraction sounds insane.

I worked on a new internal PKI, nothing as high stakes as banking, but we did have a security consultant who had been around the block. We did, of course, purchase such HSMs and design a signing ceremony. But in his opinion it was also normal and expected to keep a decryptable copy of the private key, in case of e.g. changing HSM vendors. It would be even less accessible, but it would exist.

That's a good point - I guess if you know what you are doing then I guess it's fine.

One article said they key was left in clear on a laptop, so maybe a HSM migration was involved.

Interesting.. What Open source tools do you use for this? I would love to read further how I can not have a printed copy of a master key in a safety deposit box.

For card system master keys, you don't use open source tools, you buy a set of properly hardened hardware modules, and follow the appropriate (documented, tested, verified) 'rituals' on them. IIRC the set we used cost something in the ballpark of $50k-$100k, which is not really much compared to all the other things that are table stakes of doing it properly. You can do it much cheaper if the risks are lower and you need less tamper resistance and auditing because you're less likely to have (for example) one of the trusted authorised employees be malicious and willing to invest nontrivial effort in circumventing the system.

Because losing these keys can be very, very, very expensive.

For an example (somewhat similar to this Postbank case) see India Cosmos Bank 2018 incident (https://www.reuters.com/article/cyber-heist-india-idUSL4N1V5... is one link) where criminals generated fake cards to cash out some $13 million; and replacing 12M bank cards also has an huge cost to replace the cards (perhaps roughly $12M - $1 per card replacement is plausible though possibly on the cheap side) even if we ignore the reputation cost.

Here is an implementation of this:

https://www.cloudflare.com/dns/dnssec/root-signing-ceremony/

They have a page for each signing ceremony:

https://www.iana.org/dnssec/ceremonies/41

With a script of everything done:

https://data.iana.org/ksk-ceremony/41/KC41_Script.pdf

You usually use your HSM vendor tooling, in the KCs I was involved the backup secrets were not only split between multiple holders but each holder held its part on a PIN protected smartcard.

We keep printed keys in a safe. I suppose if we were a bank we'd do it differently. They're in "tamper evident" envelopes, too. These are AWS master keys, Bitlocker encryption keys, etc.