Currently, aggregator projects work with HN because they know and understand the HN markup specifically. In an ideal world (and one in which we don't live, obviously), a "scraper" library should be able to identify things like comment streams based on contextual information. Think of the power that comes just from having indexes that are able to identify the title and content body. Now, what if we take that a step further and build an indexer that can recognize comments. One that can infer that one comment is made in reference to another based on its nested hierarchy. Are tables the right structure for that?
I'm asking you to dream. I'm asking you not to be complacent with the tools that "work" today. That's all. If you're content to use what you've got, and you don't care if we ever end up with markup that enables these powerful new ways of relating to data, fine, but don't say it doesn't matter. It matters.
There are a number of aggregator projects that various HN members have built by scraping this "broken" html and they seem to work quite nicely.
I spent a couple of years deep in the SGML world and am fully cognizant of all the arguments about how content needs to be completely separated from presentation. HTML is really a weak sister in that world. I don't think my view of HTML is incorrect.
In the real world, the ship of requiring correct HTML from a gramatical perspective left the harbor back in the 90's. If what you say were true, browsers would refuse to render broken HTML.