Web servers have always had to be written for the web clients that existed, not how they should have existed.

Sadly, a 100%-compliant web server (or client for that matter) would be greatly hobbled as to who it could communicate with on the real web. Although I do think the situation is far better now than it was 20 years ago.