Do you mean Xorg rather than X11? X11 is just a protocol (or a set of protocols, depending on how you look at it). Xorg, on the other hand, is the most popular implementation. There are (were?) others, some of them much smaller in terms of source code (but also features, platform support, etc.).