Aside from specialized distros, i don't see why they have to choose only one or the other instead of providing both. Most distros - especially the Debian derivatives - have a ton of alternatives for projects that do more or less the same thing and a bunch of obscure projects, having an X server surely isn't that much of a deal.

Of course those that provide a default DE or whatever out of the box will most likely need to choose a default setting, but that is just that: a default setting. Just because my openSUSE Tumbleweed installation came with KDE doesn't mean i had to stick with it instead of replacing it with Window Maker.