It is strange how people get blinded by technology and forget about oldest, most reliable methods.
I have worked as a security officer at a credit card acquirer (basically, processing credit card transactions between terminal and the bank).
The process to safeguard your PINs is extremely complex but at the very end relies on pieces of paper with passwords and key components written down, put in tamper evident envelopes and then into safes.
In case of HSM keys, the HSM itself is built to loose those keys at the least pretense. It is also not possible to retrieve the keys.
So whenever you want to provision a new HSM or even just move it couple centimeters (it looses keys when you try to move it) you have to go to the components written on paper.
It is just like a password on a website -- you have to be entering it regularly. But once you enter the password you don't need to retype it for every HTTP request.
Key components = password HSM = browser Message with PIN block = HTTP request