Cool info. Upvote.
As far as some of the issues you raised like battery replacement, I would treat them as disposable. In the next few years a complete computer will be available for so little that we will consider it disposable. Prepare a server once, enclose it, make sure the only way in or out is an SSH connection. When it's time to set up a clone to replace it, clone everything via an SSH session and trash the first server.
That could work, but the advantage of an HSM is that the key material is physically prevented from leaving the device, barring acid/x-rays/other attacks. All encryption happens on the device itself. With an open ssh connection you could still remove the key material from computer.
There's a whole chapter dedicated to HSMs in Security Engineering [2], which is available online. There are clever ways to attack them, and yes, the booby trap idea has been done, typically by using something light-sensitive. I'm not aware of any concrete-encased HSMs, however...
It's an interesting topic. That are lots of challenges around them too. It will probably have a battery backup, so how do you allow someone to replace the battery without wiping the keys? Or can only people with access to the keys replace the batteries? That won't work if you're doing mathematical secret sharing, however, since there's no physical way to do that.