This is a good idea, however one drawback is that Home Assistant sometimes needs "physical access" to devices connected to the computer that it's running on, for example Bluetooth and Zigbee dongles. Of course you can pass through such access to a virtual machine, but it does become somewhat nontrivial.

Fwiw it is rather trivial and reliable to passthrough individual usb devices in Proxmox - point and click even