News for hackers. Stuff that matters.
Re:Ask HN: Why am I not permitted to crawl HN? by VierScar on Monday May 20, 2013 @01:00AM (#5735990)
Firstly, read the robots.txt page (https://news.ycombinator.com/robots.txt), it states a crawl-delay of 30 seconds so don't crawl faster than that. Secondly, obey the robots.txt page. Don't crawl pages listed in their Disallow rules.
Re:Ask HN: Why am I not permitted to crawl HN? by mooism2 on Monday May 20, 2013 @04:21AM (#5736527)
TIL robots.txt has a crawl-delay directive. Thank you.