Re:Ask HN: What are best tools for web scraping?
by sharmi on Tuesday November 14, 2017 @09:11AM
(#15695073)
Nope. It is very specifically tailored to crawling. If you just need something distributed why not check out RQ [0], Gearman [1] or Celery [2]? RQ and Celery are python specific.
[0] : http://python-rq.org/docs/
[1] : http://gearman.org/
[2] : http://docs.celeryproject.org