Crawler/Scraper bot recommendations please
Hoping someone can point me in the right direction please: Is there a good fast bot that will pull down websites from a large list? I want to search them offline to match website text against another database (offline because Im assuming it will be faster and nicer for the websites I'm searching).Thanks! :)