News for hackers. Stuff that matters.

Ask HN: Why am I not permitted to crawl HN?

I was trying to crawling all articles on HN to do some data analysis stuff just for fun. But it seems like my IP is banned after I've crawled a few hundreds pages(at a very low rate of a few seconds per page).

I believe it would be a good thing if HN allow people to get its data just for some fun side-projects, or even better make it publicly available. It would encourage the hacker culture to just do some fun and innovative things.

What do you guys think?

( 5 comments | 13 points | on HN )
Re:Ask HN: Why am I not permitted to crawl HN? by VierScar on Monday May 20, 2013 @01:00AM (#5735990)
Firstly, read the robots.txt page (https://news.ycombinator.com/robots.txt), it states a crawl-delay of 30 seconds so don't crawl faster than that. Secondly, obey the robots.txt page. Don't crawl pages listed in their Disallow rules.
Re:Ask HN: Why am I not permitted to crawl HN? by mooism2 on Monday May 20, 2013 @04:21AM (#5736527)
TIL robots.txt has a crawl-delay directive. Thank you.
Re:Ask HN: Why am I not permitted to crawl HN? by rcsorensen on Monday May 20, 2013 @01:14AM (#5736028)
Re:Ask HN: Why am I not permitted to crawl HN? by orangethirty on Monday May 20, 2013 @12:52AM (#5735964)
I think you should understand things from the POV of YC. Letting people crawl the site at will would force PG to scale the site to a bigger size. For what? So you can get your free data?
Re:Ask HN: Why am I not permitted to crawl HN? by jcla1 on Monday May 20, 2013 @02:41AM (#5736247)
Here's a guide on how you are able to get HN data fast without being banned: http://jcla1.com/blog/2013/05/13/crawling-hackernews/ Disclaimer: It's my own blog.