The difficulty of implementation is at most half the problem, IMO. Driver vendors can afford to maintain a 'merged' version of the spec with extension diffs applied, and can pay experts to acquire and retain knowledge on the breadth of the spec.

The challenges arise in validation and actual development. Validating that a driver works correctly is VERY difficult due to the complexity of the spec, and you can't really afford to have a huge test team with as much experience and knowledge as your driver development team. Even once you've validated and shipped your driver, you can't know how end-users are going to exercise it.

Then as a developer, not only do you have poor knowledge of the spec, but you have no knowledge of how each vendor interpreted the spec and whether or not their implementation matches their expectations.

As the surface area of OpenGL and the complexity of each entry point grow, this is only getting worse.