Not exactly. You don't want someone sneaking in and misappropriating the HSM to authorize something bad. And if you set the system up for unattended recovery from a power failure, then in all likelihood someone walking off with the server the HSM is in can use those keys indefinitely. But there are options.

Some HSMs have self-destruct mechanisms that attempt to prevent physical access to the private key (ie by lapping the chip). Some vendors (nCipher, IIRC) have a smart card (a second HSM) that is required to authorize certain activities, like signing, or key recovery. In fact they had a byzantine generals solution that either had the key or a password for the key split between n cards. In the latter case you needed one of the original HSMs in order to clone the key, so a movie plot where you kidnap the entire team at a conference doesn't work. During initial setup the cert would be generated on the first HSM and copied to the others, having never seen daylight.

That system was quite difficult to explain to users, and I had to document it just so I wouldn't get confused and trigger a reset of the evaluation hardware (at which point all of our test artifacts have to be rebuilt).

It might be more complicated to start WWIII than to protect a signing certificate, but only just.