News for hackers. Stuff that matters.
Re:Ask HN: What are best tools for web scraping? by frik on Tuesday November 14, 2017 @11:56AM (#15696779)
> We use Redis to send task (update / discovery) to our crawlers.

Some kind of queue implemented with Redis? How does it work?

Re:Ask HN: What are best tools for web scraping? by samtc on Tuesday November 14, 2017 @02:07PM (#15698140)
It's a simple redis list containing JSON task. We have a custom Scrapy Spider hooked to next_request and item_scraped [1]. It check (lpop) for update/discovery tasks in the list and build a Request [2]. We only crawl max ~1 request per second, so performance is not an issue.

For every website we crawl we implement a custom discovery/update logic.

Discovery can be, for example, crawl a specific date range, seq number, postal code.... We usually seed discovery based on the actual data we have, like highest_company_number + 1000, so we get the newly registered companies.

Update is to update a single document. Like crawl document for company number 1234. We generate a Request [2] to crawl only that document.

[1] https://doc.scrapy.org/en/latest/topics/signals.html

[2] https://doc.scrapy.org/en/latest/topics/request-response.htm...

Re:Ask HN: What are best tools for web scraping? by thibaut_barrere on Tuesday November 14, 2017 @12:28PM (#15697144)
See https://sidekiq.org for instance.
Re:Ask HN: What are best tools for web scraping? by CGamesPlay on Tuesday November 14, 2017 @12:14PM (#15696987)
Probably not what the GP uses, but Resque does this in Ruby land.
Re:Ask HN: What are best tools for web scraping? by bdcravens on Tuesday November 14, 2017 @05:21PM (#15699743)
Sidekiq has emerged as a better option to Resque