News for hackers. Stuff that matters.
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Wednesday June 15, 2011 @05:03PM (#2659014)
Our default crawler obeys robots.txt and it looks like the /results URLs are not allowed. However.. I think you could start from a URL like http://www.youtube.com/watch?v=sAzhOSbxMiI and then crawl to the linked videos from there...
Re:IndexTank / 80legs Crawlathon (developer contest) by revorad on Wednesday June 15, 2011 @05:16PM (#2659053)
Oh I didn't know about the robots.txt rule for /results. Thanks for pointing it out or I would have built my own crawler and got banned. I think I'll go with playlists.