The keywords you are looking for are PCI-passthrough, KVM and VT-d. It is possible and performance is as great as it sounds, but it does not work on every hardware. https://scottlinux.com/2016/08/28/gpu-passthrough-with-kvm-a... is a howto. I did not do this myself yet, would need a second gpu (I think that is about to change, but support for shared passthrough gpus - really not sure how that is called - is not there yet).