As I was trying to say, there are ways to make that less of an issue by moving other factors off-site. You can configure (some) HSM cards not to be available on boot, requiring a human to come in and reactivate them with a password or a fob that they bring with them, and leaves with them.

Then your biggest problem is people thinking that stealing the cards will get them anything. Which, they're not entirely wrong, because those things are damned expensive. So you need a 'kinda' secure facility.

Again, the issue is that any hardware that will start up for you without any action on your part will likely start up for anybody else, too. Your laziness will probably win out...