Further up the stack, ARM and some other embedded chips commonly have dedicated AES instructions that operate on a 128-bit or 256-bit non-volatile register that cannot be directly read. They include a generate instruction that sets a write-once register[1] to a random value--hopefully using a hardware RNG. This effectively creates a unique key that cannot be cloned but can be used by software to be build more complex primitives (hardware attestation, exporting secret keys off the device, etc). See, e.g. https://www.xilinx.com/support/documentation/sw_manuals/xili...
"Web scale" HSMs don't usually implement the strongest method discussed above. You need to be able to export ("wrap") and load ("unwrap") an arbitrary number of keys, perform a high rate of operations, and handle significant throughput. They may use something like the non-cloneable AES key for wrapping and unwrapping, but for whatever reason (hardware cost, development time, laziness, ineptitude, lack of demand) everything else is usually controlled by software, with the unwrapped secret keys visible to the software. Sometimes this software runs on a separate chip or within a different privilege domain (e.g. TrustZone); sometimes not, as I presume is the case with the hacked Gemalto HSM device. But in any event it's a distinction without a difference--it's a pile of buggy software when not even a single line of potentially buggy software should be able to read the keys. In principle they could use formal verification techniques, but apparently not many do. They don't even seem to make use of projects like seL4, which would get them 80% of the way there.[2]
Anyhow, it can be done properly. And some HSMs may do it properly. It's hard to tell because the industry is so opaque. You're right to be skeptical, just don't assume nobody is doing it properly, though it's probably true very few are.
[1] These registers are often implemented as so-called eFuses--https://en.wikipedia.org/wiki/IBM_eFUSE
[2] Hardware can have bugs, too. The best way to approach the problem is probably to find the right mix of hardware and software that permits you to formally verify most if not all the critical components and semantics. Again, seL4 is an obvious choice here as a core architectural component; it's outright negligent seL4 or something with similar guarantees isn't more commonly used.
EDIT: grammar