Re:IndexTank / 80legs Crawlathon (developer contest)
by jdrock on Thursday June 16, 2011 @02:31PM
(#2662558)
True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.
[ Parent
| on HN ]