Re:IndexTank / 80legs Crawlathon (developer contest)
by jdrock on Wednesday June 15, 2011 @05:03PM
(#2659014)
Our default crawler obeys robots.txt and it looks like the /results URLs are not allowed. However.. I think you could start from a URL like http://www.youtube.com/watch?v=sAzhOSbxMiI and then crawl to the linked videos from there...
[ Parent
| on HN ]