The keywords you are looking for are PCI-passthrough, KVM and VT-d. It is possible and performance is as great as it sounds, but it does not work on every hardware. https://scottlinux.com/2016/08/28/gpu-passthrough-with-kvm-a... is a howto. I did not do this myself yet, would need a second gpu (I think that is about to change, but support for shared passthrough gpus - really not sure how that is called - is not there yet).

I tried it two years ago. I couldn't tell the difference performance wise. A large majority of the games in my (extensive) stream library worked. Some wouldn't stream to my Steam Link [1] (black screen). Couldn't get controllers to work on a few of them. One or two didn't have sound. One in particular would crash and somehow make the VM unbootable. Made two attempts with that game and ended up giving up after that.

The host system was Arch, the guest was Windows 7. Fedora is the only other distro I ever heard being it done on. This page [2] was very helpful.

[1] http://store.steampowered.com/app/353380/

[2] https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVM...

There's a subreddit dedicated to that: r/VFIO

Seems to be a pretty detailed article: https://davidyat.es/2016/09/08/gpu-passthrough/

it works perfectly, around 97% of the native performance (according to 3dmark)

I've been using mine for almost 2 years now, the only problems I had was laggy/glitchy sound (which was fixed by an extra $20 soundcard and a line-in cable)