News for hackers. Stuff that matters.
Re:Hacker News search by beatgammit on Sunday December 09, 2012 @01:59PM (#4895611)
You can always get around this by throttling your web crawler. It will take a much longer time, but at least you'll be able to read HN in the meantime.
Re:Hacker News search by alphast0rm on Sunday December 09, 2012 @03:37PM (#4896026)
The tricky thing when doing this is knowing what rate to stop at without getting permanently banned. I built an Android Market crawler two summers ago, and luckily Google only temp bans (from my experience), so that might be an easier project without any risk.
Re:Hacker News search by quesera on Sunday December 09, 2012 @09:59PM (#4897254)
Respecting robots.txt is probably the best plan.
Re:Hacker News search by ig1 on Sunday December 09, 2012 @05:56PM (#4896537)
Use disposable IPs.