You need to use either the built in Mac client or something like Remotix if you’re using Windows. Every other vnc client I’ve tried performs terribly as you say.