> Xorg is a network system, meaning it is very inefficient. (Imagine using TCP to talk to your SSD)

I am quite sure that there is something wrong with your system if your X clients run through the IP stack by default.