There's security, basically there's two vectors a Hetzner compromise can be used against us, either they reboot a machine and get our data through rescue mode, or they gain physical access. VPS'es can be hacked fully digitally and transparently (i.e. if you have access to the hypervisor you're in).

There's performance concerns as well, but those don't really apply for ARM as obviously you can do better.

Performance is much better compared to VMs. Lower overheads, no emulated devices, and memory and storage controllers are not shared with other VMs. If you don't need "elastic" scaling, i.e. you have a relatively constant (base) workload, bare-metal dedicated servers should have a much higher performance/price ratio.

Compared to VPS, you give up some flexibility (bare metal servers are generally only available for a month), but, at equivalent price, the specs are better on bare metal servers.

Colo can be even cheaper, but it needs additional skills, and more upfront money investment. It's a whole different world.