I think I failed to explain the crucial part of the argument so let me try from another viewpoint: OEM like NVIDIA do not make their drivers FOSS because it makes device-specific features and improvements known. To make something work well for the device as well as FOSS, they would have to publish the software instructions specifics. That sort of things enables competition to better understand their silicon and thats exactly what they do not want.

In an ideal world, libre is fine and serves all devices well. But with commercial interest in play and so many differentiation in features, it is hard to imagine libre drivers being able to cater to all device optimally