I agree, in my experience VNC has given better performance than X through the standard network transparency feature.
If you have the need for a remote desktop setup in the future, I'd recommend checking out NoMachine. It's closed-source, but from what I've seen so far it offers better performance than VNC.
It's a bit like Mapbox: you've got a database full of semantic geometry state that applications can poke (the X server), and then you've got a fronting daemon that renders, serves, and caches graphical "tiles" created by querying that geometry (the VNC server), along with exposing a protocol through which clients can easily poke at the database by referring to it in terms of the tile-server's screen coordinates.
Also, other people might say it's clumsy, but personally I really enjoy the architecture of containerized GUI apps: you've got xvfb running in the container as an X server, and then x11vnc or the like also running in there, talking to xvfb, serving VNC "tiles" and receiving key and pointer inputs over a socket. It's like the OSI network layers: X11 is the application layer, VNC is the presentation layer.
Here's one data point: It has caused endless troubles on a digital signage project I worked on a couple of years ago. Having a proper solution then would save us tons of hours trying various workarounds to minimize the effect (while at the same time having decent performance).
On the other hand, the last time I (successfully) used X's network transparency was back in '98 on a couple of Sun workstations (when I had access -- most of the time it was VT220 instead :) Since then, every time I did try it, I ended up regretting it since the bandwidth and the latency were too poor for any sane usage (VNC ended up working way better).
Which is not to say networked X is not valuable -- just that it's misleading to think you're (or myself, for that matter) a typical user.