HSMs come in all sizes, from a chip in your phone (secure element) or even a dedicated part of a SoC chip, to a big box in a datacenter that can handle tons of requests per second.

The idea is having dedicated hardware to protect the private key material. This hardware can execute signing operations, so it can use the key but it can't share the key material itself. It is usually also physically hardened with techniques to extract said keys, like sidechannel attacks based on power draw, X-ray inspection, decapping etc.