How can you ever really be sure that there is no way to change the code running on the hardware, either unintentionally via some exploit, or intentionally via a deliberate backdoor or a debugging interface enabled in production?

As a practical example, I have never heard anyone considering the freedomness of firmware in eMMC flash memory chips. But the talk "eMMC hacking, or: how I fixed long-dead Galaxy S3 phones" from CCC reveals that actually, Samsung eMMC chips have an undocumented debug interface to read/write the RAM of the firmware running on the ARM core inside the eMMC chip.