I did this too and used kitty for maybe more than a year. I recently switched to gvim and using :terminal though; the font rendering is better and actually I think it's even faster. Give it a try.

P.S. weirdly on Linux it's entirely opposite; vim in an xterm is much faster than gvim.