I have the macOS side connected via HDMI and Windows via DisplayPort. Then I switch inputs when through the monitor's OSD for changing the OS (I think there's a way to do this through DDC/CI commands depending on the monitor, in which case you'd be able to write a script to do this, haven't looked into that yet). I have a USB switch for moving all device connections to the OS of my choice.
Also the board I have, an X299 Designare, has several USB controllers on their own IOMMU group so you can pass through each to their own VM.
I just run the following commands to stop gdm and X after booting:
sudo systemctl stop gdm
sudo systemctl stop systemd-logind
sudo killall gdm-x-session
Then start my VM. libvirt automatically unbinds it from the host and binds to the VM. You're right though, in the past this has been problematic. It is working fine though now at least on my Radeon WX 5100.I would still be using ESXI now, except my (unusual) goal was to run 10.9, and I couldn't for the life of me get GPU passthrough working in 10.9.
The neat thing about ESXI, though, is that it is officially supported by macOS. Only on Apple hardware, of course, but with ESXI the motherboard is all fake anyway, so it works identically regardless of the host's hardware. All you need to do is patch out the Apple hardware check, which is easy.
High Sierra and an nVidia card "just worked" for me on ESXI. (Sadly, Mavericks did not—I could not get the GPU to output anything—and that was the only OS I cared about—but you probably don't care.)
There's a lot more information on using KVM, so you may want to go that route instead. But of course, then then you need to use a Hackintosh bootloader like Clover or OpenCore. It was really cool having everything just install natively with ESXI.
(I briefly looked into using a single GPU for passthrough, which seems possible but involved rewriting the GPU rom and I'm not going to do that :)