Re:How to Crawl the Web Politely with Scrapy
by FussyZeus on Thursday August 25, 2016 @11:24AM
(#12359720)
It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.
[ Parent
| on HN ]
Re:How to Crawl the Web Politely with Scrapy
by cookiecaper on Thursday August 25, 2016 @03:33PM
(#12361798)
Not sure what that has to do with scraping. A desktop browser can be used to copy and paste chunks of content and plagiarize a site. We have reasonable copyright protections to protect authors against that. What we need to discharge are the unreasonable laws regarding network access. It's not all or nothing.
Re:How to Crawl the Web Politely with Scrapy
by stummjr on Thursday August 25, 2016 @12:03PM
(#12360127)
Yeah, but that's not just because of web scraping. Plagiarism has been an issue for centuries.
Re:How to Crawl the Web Politely with Scrapy
by duiker101 on Thursday August 25, 2016 @11:49AM
(#12359983)
fair enough but I don't think that's the main purpose. There are many many cases where you would want to scrape something and often people would probably be encouraged in doing so in a "polite" way if websites didn't make it hard.
Re:How to Crawl the Web Politely with Scrapy
by JamesBarney on Thursday August 25, 2016 @04:25PM
(#12362197)
Yes, or if they just provided a csv with all the data most people wanted to scrape anyway with a plain English explanation about how it can be used.