Re:How to Crawl the Web Politely with Scrapy
by stummjr on Thursday August 25, 2016 @02:39PM
(#12361331)
Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.
[ Parent
| on HN ]
Re:How to Crawl the Web Politely with Scrapy
by greglindahl on Thursday August 25, 2016 @03:20PM
(#12361686)
I was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits:
http://www.bing.com/search?q=%22We+have+been+receiving+a+lar...
I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.