News for hackers. Stuff that matters.

A comment on How to Crawl the Web Politely with Scrapy. Parent comment

Re:How to Crawl the Web Politely with Scrapy by AznHisoka on Thursday August 25, 2016 @03:44PM (#12361890)
https://www.linkedin.com/robots.txt

https://yelp.com/robots.txt

There goes all Linkedin + Yelp content from your index.

[ Parent | on HN ]
Re:How to Crawl the Web Politely with Scrapy by betolink on Thursday August 25, 2016 @05:50PM (#12362775)
What about https://www.facebook.com/robots.txt

..and medium-sized/small sites are even worse.

The irony of Facebook being a core part of all NSA surveillance programs and their terms of service including their "Automated Data Collection Terms" https://www.facebook.com/apps/site_scraping_tos_terms.php

[ Parent | on HN ]
Re:How to Crawl the Web Politely with Scrapy by greglindahl on Thursday August 25, 2016 @07:44PM (#12363334)
If you surf LinkedIn logged out, you'll see that there isn't very much information available anyway. And there's no money in people search.

Yelp was very responsive when blekko wrote them; as you can see ScoutJet has the same access as googlebot.

[ Parent | on HN ]