As far as I can tell, Nvidia has two modes of operation with the Linux community, "hostile" and "inept"
Hostile is when they do things like require signed firmware for their GPU's or try to force their will on things like EGLstreams, while everyone else is using GBM
Inept is their situation with things like the Tegra mobile platforms. They simply use nouveau instead I'm told. Even though the two GPU lines of course, share a ton of engineering. For some reason they just decided it's easier to use nouveau on that side, and only that side
It's even better than that now. For me no part of X is running as root. Last I checked the only reason anyone had X as root (other than driver issues) was if they were running a graphical login manager). Since the login manager has to be up and running before any users are logged in, and likely requires X, it makes since that it's got to run in a privileged context, and works most easily as root. In my case I just use xinit to start my graphical sessions after I've logged directly into a (getty) terminal.
I think supposedly GDM can do rootless X, but I haven't tested that.
Every other driver on Linux has a kernel part and a userspace API accessed through /dev devices or syscalls. It was always unclear why X needed to be any different, but I think this was more about implementation than something fundamental to X. Of course "X" is a lot of things to a lot of people, and it depends whether we're talking about the server, the protocol, the client, the extensions, the window manager, the widget library, etc.
That is not quite accurate, although it isn't far off. More accurately; the design mistakes that make/made X11 terrible were being enforced at the driver level.
Take the fact that for many years X was being run as root on linux. Horrific state of affairs for security. Everyone knows it is bad.
Some bright spark tries to write a new window system that runs as an unprivilaged user; and ran smack-bang into the fact that the drivers live in the window system because the Kernel doesn't accept closed source modules and the graphics vendors only support X.
That eventually got fixed with the Intel/AMD graphics graphics open sourcing 2008-2018; at the moment X is becoming a very thin comparability layer for most people and a mandatory pain for Nvidia users as far as I know.
There were a lot of issues like that, and still are with Nvidia. The point is that it isn't replacing X that is hard. The issues is coordinating with Nvidia is hard.