News for hackers. Stuff that matters.

A comment on Ask HN: What are best tools for web scraping?. Parent comment

Re:Ask HN: What are best tools for web scraping? by stoneridge on Tuesday November 14, 2017 @09:17AM (#15695140)
Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash.

[0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-...

[1] https://splash.readthedocs.io/en/stable/index.html

[2] https://github.com/scrapy-plugins/scrapy-splash

[ Parent | on HN ]
Re:Ask HN: What are best tools for web scraping? by PaulHoule on Tuesday November 14, 2017 @05:24PM (#15699769)
HTMLUnit in Java is a good browser emulator and can be used to work JavaScript-heavy web sites, form submission, etc.
[ Parent | on HN ]
Re:Ask HN: What are best tools for web scraping? by maxisme on Tuesday November 14, 2017 @02:15PM (#15698201)
Reading this from my phone looked like you meant there was a web scraping tool actually called “this[0]” which would be a cracking name.
[ Parent | on HN ]