News for hackers. Stuff that matters.

A comment on RoboBrowser: Your friendly neighborhood web scraper. Parent comment

Re:RoboBrowser: Your friendly neighborhood web scraper by aexaey on Thursday May 12, 2016 @04:30AM (#11682115)
It doesn't. To scrape (or fake-API) js-only websites you have to either:

- drive a browser (firefox/chrome) via already mentioned here selenium/webdriver (potentially hiding the actual browser window into a virtual X by wrapping the whole thing with xvfb-run),

- or use one of the webkit-based toolkits: phantomjs [1] or headless horseman [2].

There is also an interesting project that combines the two, i.e. it drives a Firefox (or, more precisely, slightly outdated version of Gecko) to emulate a phantomjs-compatible API. [3]

phantomjs/slimerjs are pretty popular and even have tools that run on top of them, such as casperjs [4], that geared more to automated website testing, but can be quite good at scraping or fake-APIing too.

[1] http://phantomjs.org/

[2] https://github.com/johntitus/node-horseman

[3] https://slimerjs.org/

[4] http://casperjs.org/

[ Parent | on HN ]
Re:RoboBrowser: Your friendly neighborhood web scraper by brynedwards on Thursday May 12, 2016 @04:42AM (#11682146)
I recently wrote a browser-driven scraper using Nightmare[1], which uses Electron under the hood. Another option for those who prefer python is dryscrape[2], although I haven't tried it.

[1] https://github.com/segmentio/nightmare

[2] http://dryscrape.readthedocs.io/en/latest/

[ Parent | on HN ]
Re:RoboBrowser: Your friendly neighborhood web scraper by nikolay on Thursday May 12, 2016 @01:57PM (#11685399)
Dryscrape is really cool! Thanks for sharing!
[ Parent | on HN ]