funny you say that, because TPMs aren't actually mandated to be tamper resistant, only tamper evident[1]. what this means is that you won't be able to extract the keys without destroying the device, but if you delid the chip and probe it, you can probably extract the keys. I suspect it's the same with other HSMs you see in everyday life (smart cards, smartphone with trustzone, etc.).
[1] sorry i don't have a better source: https://media.ccc.de/v/32c3-7343-beyond_anti_evil_maid
While there are some smart cards and smartcard-like HSMs (Fortezza comes to mind, but it uses the battery primarily for integrated RTC and seems to not contain any tamper detection mechanism) with integrated battery, common smartcards does not have battery.
The term "smart card" is frequently misused in popular nomenclature. As a technical term, it refers specifically to contact or contact-less (like NFC) cards with an embedded chip which are, at minimum, physically tamper resistant. For example, a typical credit card is not a smart card. A SIM card is a smart card (or token).
Tamper resistant does not imply tamper proof, which can also be a source of confusion.