News for hackers. Stuff that matters.
Re:Colly: Fast and Elegant Scraping Framework for Golang by weberc2 on Thursday October 05, 2017 @11:59AM (#15410205)
You'd need to parse and run the JavaScript, including a virtual DOM. You'd also need to support any browser events that the page depends on. It's conceptually simple, and getting support for 80-95% of cases is probably easily doable, but I imagine a long tail of cases where your browser emulator isn't quite compatible in an important way. I know Google and others do this to index the web, but in not sure about any existing open source projects.

There are no comments on this comment yet.