Imagine how good things would be if the drivers were open source and in the kernel. We would still have bugs but at least it would be possible to fix them.

Aside from the massive maintenance effort: what is keeping the community from taking all the driver code from the tons of official and unofficial code dumps and bringing them to mainline?

You can't just drop leaked code in to the kernel due to legal reasons. And even if the vendor does provide an open source dump of the source you still can't just drop it in to the kernel because it will not meet the code quality standards for linux. Vendors just hack it until it works and call it a day since they don't have to worry about unmaintainable code if they never plan to maintain it.

You definitely can drop crappy drivers to the kernel devs for later improvement as somebody else's problem. It is called "staging".