I use VT switching all the time, on a desktop with an nvidia card and a laptop with intel / nvidia, and I don't have any of your issues. I'm on arch linux so with recent kernels, systemd, etc... if that can help but I don't remember ever having them..

I don't use GDM however.