News for hackers. Stuff that matters.
Re:How to Crawl the Web Politely with Scrapy by duiker101 on Thursday August 25, 2016 @11:49AM (#12359983)
fair enough but I don't think that's the main purpose. There are many many cases where you would want to scrape something and often people would probably be encouraged in doing so in a "polite" way if websites didn't make it hard.
Re:How to Crawl the Web Politely with Scrapy by JamesBarney on Thursday August 25, 2016 @04:25PM (#12362197)
Yes, or if they just provided a csv with all the data most people wanted to scrape anyway with a plain English explanation about how it can be used.