If possible, it's nice to keep an offline copy of your key material too. Maybe locked in a safe, gpg encrypted or something.
To restore the key, you need to bring 3 out of 6 persons to the so-called key ceremony, and each has to bring his smartcard and his PIN.
The same mechanism can be used to provision multiple HSMs with the same key material. But there are other means to do this. As soon as two HSM share a common secret, also known as Key Sharing Key, they are able to exchange all key material they possess in a secure manner.
Some HSM don't even bother to store the keys they generate within the bounds of their hardware. Only it's master key is stored in it's hardware, any other key is encrypted with the master key and stored on a shared filesystem.
If this sounds artificial to you, let me assure you that such procedures are in place at various companies who deal with raw credit card data, at least in Europe. The EMV committee, the PCI organization and each issuer of credit card do mandate such procedures.
And they are very strict. We once had to ship HSMs back to the vendor, because at some point they were not supervised by at least two persons. (At least the documentation thereof was missing.)
So, yeah, the problem is political in a way that everyone is coming with their own agenda into it, which has little grounding in reality, yet affects decisions of many people substantially.
Locking your data to your hardware raises the question of what would happen if the hardware failed? Also at first glance this seems to introduce difficulties with scalability across multiple machines. Also it might make it difficult to switch between infrastructure providers.
The cost of this approach should be mentioned as a footnote.
Maybe the better solution is for society to support more small tech companies with smaller user bases that have fewer dissatisfied rogue employees to leak hashed passwords in the first place.
The root of the problem is not technical, it's political.