Scrapy Tips from the Pros

Re:Scrapy Tips from the Pros by inovica on Wednesday January 20, 2016 @09:17AM (#10938172)

Quick question. Could you use Scrapy to specific individual pages from thousands (or millions) of sites, or would you be better off using a search engine crawler like Nutch for this? I want to crawl the first page of a number of specific sites and was looking into the technologies for this.

[ Parent | on HN ]

Re:Scrapy Tips from the Pros by sheraz on Wednesday January 20, 2016 @10:50AM (#10938842)

Yes, if you use subclass CrawlSpider, then you will be able to set rules on your crawls [1]

[1] -http://scrapy.readthedocs.org/en/latest/topics/spiders.html?...

[ Parent | on HN ]

Re:Scrapy Tips from the Pros by djm_ on Wednesday January 20, 2016 @09:42AM (#10938361)

Yes. Each spider in Scrapy has a "start_urls" parameter/method, so you'd just need to fill that up with all your domains and make sure the spider has freedom to crawl across domains. Each URL would be accessed, you'd do whatever you want to do and when the spider has visited them all, it would quit.

[ Parent | on HN ]

Re:Scrapy Tips from the Pros by ddebernardy on Wednesday January 20, 2016 @01:27PM (#10940177)

> would you be better off using a search engine crawler like Nutch for this?

Fwiw we recently met a client who tried Scrapy + Frontera vs Nutch, and their assessment was that Scrapy + Frontera is twice as fast. Here's a deck on Frontera FYI:

http://www.slideshare.net/scrapinghub/frontera-open-source-l...

Aside: don't hesitate to get in touch with our sales team if you need help. We're experienced in this type of project.

[ Parent | on HN ]

Re:Scrapy Tips from the Pros by stummjr on Wednesday January 20, 2016 @09:34AM (#10938307)

Scrapy can do the job, for sure. We use it to crawl more than 2 billion pages a month. :)

[ Parent | on HN ]