Many HSMs also add advanced authentication capabilities, such as M-of-N access control and/or hardware authenticators (e.g. you need 3 of 5 smart cards to use the HSM). The other key feature usually found in HSMs but not smart cards is backup/cloning without exporting the key (in PKCS#11 terms). This means that the key can be moved between HSMs with all the protections in place. I've yet to see a smart card that does this.
How does this work? Can an attacker buy an identical HSM, back up the key, and restore it onto the new HSM?
https://news.ycombinator.com/item?id=12069784
The NitroKey HSM requires all previous setup (DKEK?) passwords and PINs. Anecdotal random unnamed vendor? Not so much:
They did, with undocumented commands, export the key from the device in an unencrypted format and loaded it into the other model
And this is exactly my point, it's all functionality which the manufacturers have decided some customer might need. I don't want that, I want a secure general-purpose Turing-complete execution environment which gives me full flexibility in what crypto, mechanisms and policy to implement. This entire industry approach is silly.
Using them from code is like using a remote API, you can't execute code, you can only make requests and receive responses, and a lot of thought it put into the interface to ensure it's as secure as possible.
As far as the physical interface, they are more like an appliance than a server, in that you don't get the normal ports you would on any other machine, the actual casing is typically alarmed so that keys are wiped when the device is opened, important circuitry is embedded in epoxy so that it's essentially impossible to attach debugging instruments without destroying it, etc.
I can't really emphasise enough how these machines are designed for security.
For instance the YubiCo HSM is just a USB device that plugs into a server.
An HSM is typically a 1-2U server, that is designed to provide high throughput of cryptographic operations. It is ultimately a collection of a few high performance servers networked together, with some custom ICs - not just a small chip. As a result, you pay up to tens of thousands of dollars for one, because it's a piece of critical infrastructure that is made to high tolerances. It's akin to buying hardware load balancers or firewalls appliances.
In addition to this, the validation process of an HSM is long. An HSM company will likely have teams of hardware engineers, software engineers, and specialised cryptography teams. There are audits for things like FIPS compliance, as well as extensive pentesting by external companies. All of this is expensive, to create a device that will never be mass market.