HSMs allows to have key stored with the option to disable key export. This means every cryptographic operation must be done by the HSM (commonly through pkcs 11 API).
HSMs have backup features and the data cannot be restored without a secret split among multiple secret holders (like https://en.m.wikipedia.org/wiki/Shamir%27s_secret_sharing). So a backup can be done on physical media and put in a safe.
It's a lot of things about key management, HSMs, pkcs interfaces to learn though.
I somehow assumed that real, non-toy HSMs involved dedicated generators as companion devices with heavily reduced attack surface, which generate and store private key material, are able to transfer it to the HSM proper (and to a paper backup), and are strictly kept offline after that.
Sometimes the world feels disappointing.