Re:IndexTank / 80legs Crawlathon (developer contest)
by revorad on Wednesday June 15, 2011 @05:16PM
(#2659053)
Oh I didn't know about the robots.txt rule for /results. Thanks for pointing it out or I would have built my own crawler and got banned. I think I'll go with playlists.
[ Parent
| on HN ]
There are no comments on this comment yet.