>> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded.
>> If you'd like to see why, read the HTML 5 parsing portion of the spec.
> Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec
I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?
HTML was a massive success, while at the same time when we tried having a strict spec with XHTML, we failed miserably.
I mean, the purist in me wants strictness but it seems lousyness wins.
The explosion of the web happened in no small part because of how easy it was to write some HTML and get a basic, working webpage out of it. If you nested some tags the wrong way and the browser just put up an error page, rather than doing a (usually) pretty good job figuring out what you actually meant, people would get frustrated faster and not bother with it at all.
But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.
The difficulty in drawing a line in the sand and sticking to the spec, though, is that of user blame. Let's say you implement a spec perfectly -- even if you are the originator of the spec -- and then someone comes along and builds something of their own that writes out files that don't conform to the spec. Your software throws up an error and says "invalid file", but the other piece of software can read it back in just fine. Users don't know or care about specifications; they just know that your software "doesn't work" for the files they have, and the other software does. If you try to tell them that the file is bad, and the other software has a bug, they really won't care.
That's how Perl ended up the way it is.
It also shows the harmfulness of binary black and white thinking in engineering. There are choices other than "just let everyone do whatever and hope all the different things picking up the pieces do it in more or less the same way" and "rigidly specify a spec and blow up the universe at the slightest deviation". Both of those easy-to-specify choices have excessive costs. There is no escape from hard design tasks. XHTML may always have been doomed to fail, but that is not to say that HTML had to be allowed to be as loosey-goosey as it is, either.
Had a gradient of failure been introduced rather than a rigid rock wall, things very likely wouldn't have gotten as badly out of hand as they did. If, for instance, a screwed up table was specified to deliberately render in a very aesthetically unappealing manner, but not crash the entire page the way XHTML did, people would have not come to depend so much on HTML being sloppy. The resulting broken page would still be somewhat usable, but there would have been motivation to fix it, rather than the world we actually live in where it all just seemed to work.
If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.