I've also been doing this and in 99% of cases the performance appears identical to me. You just need to have an extra keyboard/mouse available so that you can reduce latency.

There's one particular game lately, however, that I've been booting to Windows directly for. Assets seem to load extremely slowly even if I pass the entire hard drive directly to the VM. There might be a tweak for getting that to work but I don't really know what the source of the slowdown is.

It depends on how you set it up. If you don't isolate your CPU cores, you'll notice VM load on the host and the other way around. Other than that, the loss is negligible (unless you're going for benchmarks). I'd say in the sub-5% range, depending on the workload (GPU, for example, is usually not impacted at all). My old workstation with an i5-2500 could still play AAA games in good quality (with a somewhat modern GPU).

Surprisingly, the disk performance of HDDs and SATA SSDs is actually better in VMs for me. It seems that the caching and/or access strategy used by the Linux kernel helps a lot.

Of course there is always small cost; for one, you're running two OS at once, so not all resources are available. You'll also need to have a sound system (unless you have two sound cards or use scream, you might have latency) and if you pass-through USB via the VM subsystem instead of handing an USB controller to the VM, some high-speed devices might make problems (i.e. DJ controllers). But overall, I'd say the cost is negligible for nearly all use-cases.