Yes, the often times proprietary android linux driver for are such a pain. Not only make they it harder to reuse the hardware outside of android (e.g. in a laptop or similar). But they also tend to cause delays with android updates and sometimes make it impossible to update a phone to a newer android version even if the phone producer wants to do so.
Android did start making this less of problem with HAL and stuff, but it's still a problem, just a less big one.
As a Linux user and an ex android user, I absolutely disagree and would add that the GPL requirement for drivers is probably the biggest feature Linux has!