There are several ways to do this. The best way would be to use a router that lets you route. Create the VPN tunnel as always on. Set routing policy based on destination. Gameserver.com goes out the ISP direct, everything else goes out the VPN. You can add additional tunnels and routs as needed but the building block is having a real router that lets you set routes. Personally, I have a static VPN to work, a tor tunnel, VPN to personal offsite infrastructure and an everything else tunnel. I set routes so my traffic goes the the right place automatically, for myself and anyone else in my house.

How do you figure out what to set routing policy for? You say gameserver.com, but how do I figure that out for CSGO vs Overwatch? What about games where you're connected to a friend's computer hosting a match, such as 7 days to die or potentially minecraft?

You basically have to make a white list. It's not an easy thing to do and I don't know any way to do it dynamically.

Sometimes, the only way to make a whitelist is to see what's not working and add it to the list.

can you share more information about which router you use and possibly a link to instructions on how to set this up?

I run a custom made freebsd pfsense box, before that it was linux with iptables.

For the iptables box, I put a second 4 port intel pro 1000 nic in my normal host, ran a VM with PCI pass through so the VM controlled the nic and used that as a router.

I think there are purpose built routers out there that will do this but it will depend on the router on how you set this up. You'd have to do research on this. Pretty sure the ubiquity routers will let you do things like this. I've done similar on open-wrt.

A VPN will always add some latency; you can just turn it off when gaming

There are VPN services with gaming modes or that are built primarily for low latency gaming. This may sound ludicrous at first but if the routing between you and the game server is poor or optimized for price or bandwidth more than latency, then such a VPN service could indeed lower latency. I don't know if they do but under the right circumstances it's possible.

I'm not a networking expert, but I do think it would feasible on an open router platform to program such that certain traffic & whitelisted sites do not use VPN (e.g. your gaming traffic) whereas everything else is routed to VPN. Can't swear by it though, and seems like it may be a complicated setup / maintenance.

Seconding this. I do a lot of uploading of photos, gaming, etc. What's a realistic solution? When should I have my VPN on or off? What about when I'm on my phone on wifi? What about when I'm on my phone on data?