I was disappointed with USB for guests though, I found it impossible to restore my iPhone via iTunes on a Windows VM, because the USB device wouldn't automatically bind to the VM. Which is something even VirtualBox can do well. But maybe the tech has progressed, or there's a better solution to that issue now.
My main experience is with a Logitech g933 headset, which requires windows-only software to control certain parts of the device.
Note that modprobe configuration is important, as it assigns stub drivers to the GTX that allows it to be handed on demand to the VM.
Note that this is made for a debian 10 system (kernel 4.19), other distributions might have different requirements.
As for running the GTX 1070 on the host when the VM is not running, I'm not sure if it is possible.
On Windows it's (1) install steam (2) play. To develop with my headless freebsd dev machine, it's just (1) install ssh (2) install x11 client (3) open term.
I made some tweaks based on level1 forums information, but most of that is auto generated by a GUI.
I've been wanting to try out linux gaming and this makes it more interesting.
I've also been thinking about videoconferencing/etc on linux running in the same manner.
Also you may need hardware, especially motherboard that supports VFIO groups well.
Fortunatly there are quite big communities like Reddit in this topic. Just google "VFIO"
Maybe it is not perfect sandbox for running malicious code, but surely it is enough to make games and anti cheat systems think they are running on a real computer.
I've been doing this since 2017. Back then I had built my first desktop computer powered by ryzen 1700 + 2 GPUs.
Creating this setup had some challenges, but after the initial hurdle everything has been working perfectly (I imagine things must be easier nowadays).
Created a windows 10 VM, assigned 4 of my 8 cores, 16 GB of RAM (got 64 on the host) and a GTX 1070. This setup is enough to play pretty much any AAA game with max settings on full HD.
Besides having every game working perfectly, I get another nice side effect: sandboxing. No games (mostly proprietary software) have access to my personal/work files.