For mouse and keyboard I just use the evdev forwarding thing where you press both ctrl keys to swap between host and guest
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
With AMD cards it's relatively easy if you're willing to install two cards and have one of them just sit there doing nothing when you're not using the VM. It's also possible to use just one card and detach it from the host system, pass it to the VM, and then reattach back to the host system when you're done playing, although I spent multiple days on this and never got it working. YMMV, it was 2-3 years ago, the driver support may have improved.
With nvidia it ranges from difficult to impossible.
Obviously, it means you can't use DLSS, RT, or any other GPU-specific features, but their DirectX virtualization supports up to DX12.
It's mostly those annoying ACs with kernel modules like EAC, BattlEye, ESEA etc. that do anti VM in an attempt to prevent cheat devs from 1. debugging the AC without at least a little effort and 2. having a clean OS but reading guest RAM from the host to avoid the anticheat entirely
There's actually been a lot more of these that don't get CVEs It's one of the reasons I prefer to game in a VM with heavy network filtering and egress only through VPN
There is little to no care from game developers about security, games with actively exploitable RCEs (see pretty much the whole CoD franchise) are just allowed to stay up on Steam
Gamers are also kinda dumb and oblivious to RATs etc which doesn't help