Re:IndexTank / 80legs Crawlathon (developer contest)
by Karhan on Thursday June 16, 2011 @01:35PM
(#2662318)
Is there a good reason that reddit has banned 80legs crawling?
[ Parent
| on HN ]
Re:IndexTank / 80legs Crawlathon (developer contest)
by jdrock on Thursday June 16, 2011 @02:31PM
(#2662558)
True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.