Re:How to Crawl the Web Politely with Scrapy
by duiker101 on Thursday August 25, 2016 @11:49AM
(#12359983)
fair enough but I don't think that's the main purpose. There are many many cases where you would want to scrape something and often people would probably be encouraged in doing so in a "polite" way if websites didn't make it hard.
[ Parent
| on HN ]
Re:How to Crawl the Web Politely with Scrapy
by JamesBarney on Thursday August 25, 2016 @04:25PM
(#12362197)
Yes, or if they just provided a csv with all the data most people wanted to scrape anyway with a plain English explanation about how it can be used.