I worked on a new internal PKI, nothing as high stakes as banking, but we did have a security consultant who had been around the block. We did, of course, purchase such HSMs and design a signing ceremony. But in his opinion it was also normal and expected to keep a decryptable copy of the private key, in case of e.g. changing HSM vendors. It would be even less accessible, but it would exist.