Ask HN: What are best tools for web scraping?

Re:Ask HN: What are best tools for web scraping? by Bromskloss on Tuesday November 14, 2017 @08:42AM (#15694851)

Would you still recommend Scrapy if the task wasn't specifically crawling?

[ Parent | on HN ]

Re:Ask HN: What are best tools for web scraping? by sharmi on Tuesday November 14, 2017 @09:11AM (#15695073)

Nope. It is very specifically tailored to crawling. If you just need something distributed why not check out RQ [0], Gearman [1] or Celery [2]? RQ and Celery are python specific.

[0] : http://python-rq.org/docs/

[1] : http://gearman.org/

[2] : http://docs.celeryproject.org

[ Parent | on HN ]

Re:Ask HN: What are best tools for web scraping? by luckystarr on Tuesday November 14, 2017 @09:12AM (#15695083)

I once used it to automate the, well, scraping of statistics from an affiliate network account. So you can do pretty specific stuff, as long as it involves HTTP/HTTPS requests.

[ Parent | on HN ]

Re:Ask HN: What are best tools for web scraping? by dataslap on Tuesday November 14, 2017 @04:30PM (#15699331)

depends on the task. For example they have a decent file/image downloading middleware.

[ Parent | on HN ]