There's a whole chapter dedicated to HSMs in Security Engineering [2], which is available online. There are clever ways to attack them, and yes, the booby trap idea has been done, typically by using something light-sensitive. I'm not aware of any concrete-encased HSMs, however...
It's an interesting topic. That are lots of challenges around them too. It will probably have a battery backup, so how do you allow someone to replace the battery without wiping the keys? Or can only people with access to the keys replace the batteries? That won't work if you're doing mathematical secret sharing, however, since there's no physical way to do that.
[1] http://en.wikipedia.org/wiki/Hardware_security_module
[2] http://www.cl.cam.ac.uk/~rja14/book.htmlAs far as some of the issues you raised like battery replacement, I would treat them as disposable. In the next few years a complete computer will be available for so little that we will consider it disposable. Prepare a server once, enclose it, make sure the only way in or out is an SSH connection. When it's time to set up a clone to replace it, clone everything via an SSH session and trash the first server.
Sure, your local machine could also be raided and the keys grabbed from that, but that's not a problem with using a VPS so much as a problem with your local setup.
It's also the focus of some of the speculation about the unknown demand that triggered the Lavabit shutdown.
Sure, if the authorities targeted you specifically, they could probably pry the unencrypted keys from the server's memory (although with considerable effort it it was a dedicated or collocated machine) but widespread adoption of that model would completely stop dragnets. Only narrowly tailored investigations would be feasible, no mass surveillance.