How it works with open source drivers is that you main-line your drivers so that the kernel maintainer maintain the drivers for you, for free. Choosing to keep your drivers closed source means committing to keeping your drivers up-to-date with changes in the kernel, or writing an open-source shim that does that. Which approach is more "fragile?"
Graphics card drivers happen to be very complex beasts, somewhere along the stack they do need a compiler for multiple custom and often proprietary architectures, which most definitely no-one will maintain for free for you. There is a huge incentive to keep most of the code platform independent and only maintain a minimal kernel specific component. This part of the code base is a comparatively trivial part, essentially the kernel should get out of the way as much as possible. The kernel specific abstractions like KMS are examples of such comparatively trivial things.
Not sure what do you mean by "maintain for free". Full coverage testing is not free.