Re:Ask HN: Why am I not permitted to crawl HN?
by VierScar on Monday May 20, 2013 @01:00AM
(#5735990)
Firstly, read the robots.txt page (https://news.ycombinator.com/robots.txt), it states a crawl-delay of 30 seconds so don't crawl faster than that. Secondly, obey the robots.txt page. Don't crawl pages listed in their Disallow rules.
[ Parent
| on HN ]