Apple and Microsoft have tossed out their window managers more than once also. X is older than anything they've had.
I do, yes, which is why I avoid things like PAM and all the various FOO-kits (sadly Slackware has since succumbed to consolekit, which is why I've been moving to OpenBSD). I want to know that anything I do as my user without su or sudo or doas (depending) will not have an impact outside of my home directory.
(Though to be clear, I think a display server could conceivably be justified for a dedicated non-root user -- fbterm seems like a good model here: getty hands over the VT to a user who has write access to the framebuffer.)
Well, writing into the home directory requires hardware access, too. Where do you draw the line?
In the similar way that the kernel's VFS provides authorized access to storage hardware, the *kits (or nowadays systemd-logind) provide authorized access to video/audio hardware.
It requires access to a filesystem, which may or may not translate to writing to a physical disk, depending on the setup.
As far as the user permissions, fair enough, though I personally think hardware access should require root, for the most part.