It would have been very easy to drop all the back compat bs by saying "An HTML5 document is one that begins with the 6 bytes '<html5' and if the document is invalid, reject it. Anything else, parse however you want." Browsers that support HTML5 add text/html5 to the Accept header.
You'd still need to support HTML4 somewhere. Supporting HTML5 separately just means duplicating the common parts of the parser. The simplicity boat has already sailed.
HTML4 + HTML5 < HTML4 * HTML5Browsers are going to have a complex, ugly, recovery-enabled parser in them either way, and the effort to add HTML5 to the recovery-enabled parser isn't very big, comparatively speaking.
If you allow people to leave comments on your page or if you are serving ads, you lose a bit of control in what gets put on a particular page of yours.
In case of comments you could try and sanitize them, but with ads that's hardly possible.