News for hackers. Stuff that matters.
Re:Ask HN: What are best tools for web scraping? by samtc on Tuesday November 14, 2017 @09:41AM (#15695359)
I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service.

All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/<dataset name>/<unique key>/<timestamp>) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception.

We have cronjob (cron + jenkins) that trigger dataset update and discovery. For example, we scrape corporate registry, so everyday we update the 20k oldest companies version. We also implement "discovery" logic in all of our crawlers so they can find new data (ex.: newly registered company). We use Redis to send task (update / discovery) to our crawlers.

Re:Ask HN: What are best tools for web scraping? by mapster on Tuesday November 14, 2017 @05:24PM (#15699763)
Mind if I ask what info/data you are scraping and for what ends?
Re:Ask HN: What are best tools for web scraping? by frik on Tuesday November 14, 2017 @11:56AM (#15696779)
> We use Redis to send task (update / discovery) to our crawlers.

Some kind of queue implemented with Redis? How does it work?

Re:Ask HN: What are best tools for web scraping? by samtc on Tuesday November 14, 2017 @02:07PM (#15698140)
It's a simple redis list containing JSON task. We have a custom Scrapy Spider hooked to next_request and item_scraped [1]. It check (lpop) for update/discovery tasks in the list and build a Request [2]. We only crawl max ~1 request per second, so performance is not an issue.

For every website we crawl we implement a custom discovery/update logic.

Discovery can be, for example, crawl a specific date range, seq number, postal code.... We usually seed discovery based on the actual data we have, like highest_company_number + 1000, so we get the newly registered companies.

Update is to update a single document. Like crawl document for company number 1234. We generate a Request [2] to crawl only that document.

[1] https://doc.scrapy.org/en/latest/topics/signals.html

[2] https://doc.scrapy.org/en/latest/topics/request-response.htm...

Re:Ask HN: What are best tools for web scraping? by thibaut_barrere on Tuesday November 14, 2017 @12:28PM (#15697144)
See https://sidekiq.org for instance.
Re:Ask HN: What are best tools for web scraping? by CGamesPlay on Tuesday November 14, 2017 @12:14PM (#15696987)
Probably not what the GP uses, but Resque does this in Ruby land.
Re:Ask HN: What are best tools for web scraping? by bdcravens on Tuesday November 14, 2017 @05:21PM (#15699743)
Sidekiq has emerged as a better option to Resque
Re:Ask HN: What are best tools for web scraping? by CGamesPlay on Tuesday November 14, 2017 @12:15PM (#15696997)
I have a similar set up! How do you monitor for failures and deal with the scrape target changing?
Re:Ask HN: What are best tools for web scraping? by samtc on Tuesday November 14, 2017 @02:04PM (#15698118)
We monitor exceptions with Sentry. We store raw data so we don't have to hurry to fix the ETL, we only have to fix navigation logic and we keep crawling.
Re:Ask HN: What are best tools for web scraping? by Launchr on Wednesday November 15, 2017 @01:21PM (#15706231)
Sorry if it's a stupid question/example/comparison, just trying to understand better: You're storing the full html data instead of reaching into the specific div's for the data you might need? This way, separating the fetching from the parsing?

I'm a scraping rookie, and I usually fetch + parse in the same call, this might resolve some issues for me :) thanks!

Re:Ask HN: What are best tools for web scraping? by jimsmart on Wednesday November 15, 2017 @03:04PM (#15707261)
When I've done scraping, I've always taken this approach also: I decouple my process into paired fetch-to-local-cache-folder and process-cached-files stages.

I find this useful for several reasons, but particularly if you want to recrawl the same site for new/updated content, or if you decide to grab extra data from the pages (or, indeed, if your original parsing goes wrong or meets pages it wasn't designed for).

Related: As well as any pages I cache, I generally also have each stage output a CSV (requested url, local file name, status, any other relevant data or metadata), which can be used to drive later stages, or may contain the final output data.

Requesting all of the pages is the biggest time sink when scraping — it's good to avoid having to do any portion of that again, if possible.