IIRC, part of the problem is X11 forwarding works best when the software is using raw X primitives (someone can correct my terminology if I'm wrong) because then X11 can just send draw commands. Both GTK and Qt don't use the X11 primitives so instead the server has to send entire pixel buffers, which is naturally a lot slower.