I had a Nvidia Riva TNT2 and later on an Nvidia GeForce. I ran Linux on it.
I had stability issues with the driver, but I solved it the following way: ran one X server with a DE, and another X server with Nvidia's proprietary driver (mostly for games). This way, if I had to kill the X server using Nvidia's driver I didn't lose any work.
If that wasn't enough, all the bloody time there were massive security problems found in Nvidia's proprietary driver. I don't know if that is still the case, cause I switched away to ATi in the 00s, and Intel graphics cards + ThinkPad as laptop. ATi/AMD has come a long way ever since. Their FOSS drivers are stable, and they deliver (see various Phoronix benchmarks).
Many distros do this already. Both for graphics drivers (xf86-video-modesetting) and input drivers (xorg-input-libinput)
There's glamor, but AFAIK it's not as tested as it should, and is still shoehorned into old model.
An example of not following the old model is Xsgi, which was (hw) compositing and quite ingenious in many ways.
Whenever I had a chance to run on nvidia binary drivers, the only things I occasionally missed were some new features, or having to wait a bit longer to update the kernel. Stability was better, drivers more performant, and I don't remember daily fighting with memory leaks.
And X.Org's driver architecture could be replaced completely (in fact, it could be made to run on the same stack as Wayland) - it wouldn't be the first compositing Xserver around, and could use methods that would deal with noticeable to many lag involved in compositor-based UI.