Normally, firmware does not make the driver itself less open source.

I get it and I'd also like for all components of my devices to be open source. Still, the parts I really care about and have any idea of how to fix stuff in, are open source.

For nvidia devices there is (or was? It's been a while) a whole other set of horrible user-space closed-source stuff.