It works fine here on Ubuntu 19.10 (kernel 5.3).

I just run the following commands to stop gdm and X after booting:

  sudo systemctl stop gdm
  sudo systemctl stop systemd-logind
  sudo killall gdm-x-session

Then start my VM. libvirt automatically unbinds it from the host and binds to the VM. You're right though, in the past this has been problematic. It is working fine though now at least on my Radeon WX 5100.