News for hackers. Stuff that matters.
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Thursday June 16, 2011 @02:31PM (#2662558)
True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.
Re:IndexTank / 80legs Crawlathon (developer contest) by xtacy on Friday June 17, 2011 @10:20AM (#2665574)
So, 80legs caused more traffic volume to reddit than Google crawler? That seems interesting; why would it be the case?
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Friday June 17, 2011 @11:20AM (#2665825)
Not necessarily more traffic, though that's possible. More likely it was the fact that it was coming from multiple IP addresses.