I think you raised an interesting point here 'for a lot of us Xorg is not an important factor', I agree. The same could be said about some of the features that systemd provides that cause a lot of flames (binary logs). It has been said before that systemd is monolithic, and this is probably what makes switching so hard.

It is all-or-nothing, whereas if you could gradually replace the old sysvinit/policykit/consolekit/etc. stuff with systemd/logind then problems during that transition could be debugged more easily. You could also choose to not replace some components where the systemd/non-systemd replacement is broken.