Sure. But the attacker needs to actually get in, which is considerably harder than getting into a hotel room. But more relevantly, the kinds of countermeasures that get you from level 1 to a higher level don’t seem likely to help at all — if some evil-maids or otherwise fully compromises a machine hosting a FIPS 140-2 level 4 HSM, they likely get the unrestricted ability to perform cryptographic operations using keys protected by that HSM, but they get this by using the HSM’s normal API. If they can convince the HSM to export its keys to another HSM (oops) or to otherwise leak the key material, they get the key material. But this doesn’t seem like it has much to do with physical attacks against the HSM.

Now if someone evil-maid attacks the HSM itself, that’s a different story. Any good HSM should resist this, especially one found in a portable device. And this is because you can steal an entire important corporate laptop or other portable device without necessarily raising an quick alarm, whereas I have trouble imagining someone walking off with the HSM out of an IBM mainframe or with an AWS HSM without the loss being noticed immediately.

(To be fair, in the mainframe case, some crusty corporations seem to have a remarkable ability to fail to notice obvious crypto problems like their public facing certificates expiring. But a loss of an entire HSM from a secure large cloud datacenter will, at the very least, immediately trigger “elevated failure rates” or whatever they like to call it…)