Re:Hacker News search
by alphast0rm on Sunday December 09, 2012 @03:37PM
(#4896026)
The tricky thing when doing this is knowing what rate to stop at without getting permanently banned. I built an Android Market crawler two summers ago, and luckily Google only temp bans (from my experience), so that might be an easier project without any risk.
[ Parent
| on HN ]
Respecting robots.txt is probably the best plan.