Video drivers have been just fine for me for the past six years or so, but perhaps that's because I always buy nvidia cards that are well supported.
The only place OS X has a leg up on Linux IMHO is video codec support, and you can't really blame the open source developers for this one.
I keep waiting for a hardware manufacturer to issue 3 or 4 different machines and make Linux work very well on them, akin to what Apple does with OS X. Most of the issues I see people have with Linux is due to the immense variability in hardware combinations. Eliminating this variability would make it pretty easy for a company to provide a seamless desktop experience to users.
I admit ignorance--the last time I used Ubuntu regularly on the desktop was just before my first mac arrived in late 2006. I've toyed with it since then; but never gone through installation and configuration.
...and let's not even get into video drivers. :P