The W3C was too strict where it didn't really matter (the basic markup) and too loose where it did matter (everything else).
Also there was no specification right from the start and the feature set was very limited, so that everyone started implementing their interpretation of HTML and adding their extensions. Trying to combine and standardize all the things existing in the wild would necessarily be a hard task. Many early extensions were probably also quite ad hoc and even if not they most likely did not take into account interactions with extensions from others.
But in the end this could probably never have happened much differently, future developments and requirements could not known in any detail and even if nobody would ever have spent a decade to write good specifications for a new technology which might or might not get adopted.
But despite all the flexibility offered by SGML and because probably nobody implemented HTML parsing by using a SGML parser, we ended up with malformed HTML documents who's interpretation was essentially defined by whatever the ad hoc HTML parser implementations in browser did.
HTML5 finally abandoned the idea of basing HTML on SGML and a DTD and instead essentially formalized the status quo of HTML parsing and put it into the specification. At least this is my understanding as a non-web developer who gets to work with HTML only occasionally.