No normal application developer "have to deal with a network protocol". Even if you write code directly against Xlib that is largely transparent to the developer.
Performance was an issue until XRender and DRI arrived around 2000. It has not been an issue for years.
Incidentally, one of the reasons Wayland is becoming so attractive is exactly because so little of what X was traditionally is actually being used any more in most setups, since XRender and DRI combined means most apps does all or the vast majority of rendering client side, and the server is largely left as a resource manager and compositor and not much more.
I would argue that Linux isn't mainstream because it has nothing like Apple's budget, specially on design and marketing. Add to that those hardware vendors that are actively hostile to support from other operating systems, and software developers that don't care, and you have it. Still, Linux is doing fine, for what resources it has.
At the time, there were many commercial Linux distributors - Red Hat, Mandrake, SuSE all spring to mind - and any one of them could have beaten Apple to the punch in offering a Unix desktop environment that was easy to use and, for lack of a better term, pretty. They didn't, though. Why not? I think the need to build on top of X is one thing that hampered Linux desktop environments when competing with OSX.
I'm not a linux guy at all, and it saddens me when I am the one to teach (supposedly advanced and technical) linux users about X forwarding.
This works even across different architectures, and the debugger is completely network unware, is all handled transparently by the file system interface to processes.
Some goes for the window system, which also works across the network even when it knows nothing about networks.
See this paper by Rob Pike about how to design window systems: http://doc.cat-v.org/bell_labs/transparent_wsys/
On the other hand, that very separation is X's greatest hindrance. It imposes a fairly large cost, not just in terms of performance, but also in terms of complexity. You have to deal with a network protocol just to display things on the screen. No other windowing system imposes this level of cost on the developer, and it can be argued that the reason we never got a Linux desktop was because of X. Simply put, Apple showed us what could be done with a Unix system if you abandon X and set allow your GUI API to talk directly to the hardware. A large part of Apple's success can be tied directly to their decision to forego X11.