News for hackers. Stuff that matters.
Re:Colly: Fast and Elegant Scraping Framework for Golang by integrii on Thursday October 05, 2017 @01:51PM (#15411051)
You will get to initial source code with this, but if you want JavaScript to work, you just use PhantomJS. If you want PhantomJS to be usable, you is casperJS... Until you find a site with FuzeJS or some other JavaScript or html intense site. Those won't render in PhantomJS.

For that stuff, as of a few months ago, you can use Chrome headless. I wrote a couple go packages to make that easy. It basically runs headless chrome with a JavaScript REPL console you can use to interact with the session. https://GitHub.com/integrii/headlessChrome

I was also able to smash my while scraper bot into a docker container after working around a couple bugs.

Re:Colly: Fast and Elegant Scraping Framework for Golang by stcredzero on Thursday October 05, 2017 @04:26PM (#15412325)
It basically runs headless chrome with a JavaScript REPL console you can use to interact with the session.

That looks cool! Would I be able to run Node scripts?

Re:Colly: Fast and Elegant Scraping Framework for Golang by simlevesque on Thursday October 05, 2017 @05:20PM (#15412677)
> Would I be able to run Node scripts?

No. Since you can't run Node scripts on Chrome, the same is true for Chrome headless.