News for hackers. Stuff that matters.
Re:Colly: Fast and Elegant Scraping Framework for Golang by gjem97 on Thursday October 05, 2017 @09:50AM (#15409083)
The HTML parsing part appears to be "golang.org/x/net/html". Does anyone have experience parsing "real world" html with this? How does it do?
Re:Colly: Fast and Elegant Scraping Framework for Golang by jerf on Thursday October 05, 2017 @12:23PM (#15410398)
It's an HTML5-compliant parser. That means that, modulo bugs, it should produce the same results as any other modern HTML parser, which should also all be based on HTML5.

For context, since I think a lot of people are still unaware of this, the HTML5 standard precisely specifies how HTML should be parsed: https://www.w3.org/TR/html5/syntax.html#parsing This is based on a survey of how the various browsers were handling it in reality, so it's not just one of those theoretical things that everybody ignores, it's an algorithm extracted from the brutal pragmatism of many separate code bases over many years. In theory now, all HTML parsing libraries should now be able to take the same input and produce the same DOM nodes. In practice I've not used a variety of such libraries, nor have I fed them very much pathological input, so I can't vouch for if this 100% true in practice, but in theory, there should no longer be any significant differences between HTML parsers in various languages, as they come on board with HTML5 compliance.

Re:Colly: Fast and Elegant Scraping Framework for Golang by anonacct37 on Thursday October 05, 2017 @02:49PM (#15411625)
I've used it to parse alot of broken HTML for a very large company.

I have not used it to attempt to parse all the HTML on the web.

My impression is that it's pretty good.