News for hackers. Stuff that matters.
Re:Ask HN: How do you feel about web scraping? by wslh on Tuesday June 06, 2017 @06:48PM (#14502159)
I think there is a contradiction between web scraping ToS and Internet neutrality. Allowing a site to be scraped (not different with crawling nowadays) only by Google but not by others violates that principle and concentrate de power within few companies.
Re:Ask HN: How do you feel about web scraping? by crispytx on Tuesday June 06, 2017 @07:48PM (#14502564)
Yeah wtf? Everyone let's Google crawl their site but it's naughty for anyone else to do it... such bullshit!
Re:Ask HN: How do you feel about web scraping? by jmngomes on Wednesday June 07, 2017 @06:19AM (#14505001)
Probably because scraping by search engines is a transaction that benefits both sides, as search engines bring traffic to the content's source.

"it's naughty for anyone else to do it" if that everyone are companies that will be using or monetizing that content without providing any kind of attribution or compensation to the source/creator.

Re:Ask HN: How do you feel about web scraping? by tomascot on Wednesday June 07, 2017 @07:41AM (#14505396)
Except now that Google shows the info directly in the search results.
Re:Ask HN: How do you feel about web scraping? by taken__-- on Wednesday June 07, 2017 @07:52AM (#14505474)
Google isn't the only search engine.
Re:Ask HN: How do you feel about web scraping? by spynxic on Tuesday June 06, 2017 @07:54PM (#14502585)
Allowing a site to only be scraped/crawled by professionals (i.e. Google) ensures that those engaging such activity do so in a respectable manner.

It would be ideal to define the terms for scraping (similar to how API calls work) as opposed to the acceptable authors of scrapers.

Re:Ask HN: How do you feel about web scraping? by wslh on Tuesday June 06, 2017 @07:57PM (#14502601)
I am not sure you can define those terms in a formal way. You can crawl the data in some way but use it for many different purposes that are difficult to control.
Re:Ask HN: How do you feel about web scraping? by ellimilial on Wednesday June 07, 2017 @04:25AM (#14504553)
Fancy extension to robots.txt?
Re:Ask HN: How do you feel about web scraping? by madshiva on Wednesday June 07, 2017 @05:10AM (#14504733)
They are professionals now, but at the beginning it was not at all.
Re:Ask HN: How do you feel about web scraping? by spynxic on Wednesday June 07, 2017 @07:04PM (#14510559)
I would imagine even in the beginning that they, as developers, practiced good procedures while scouring the internet.

Given the difference in the size of the internet now and then, the backlash faced for poor practices may not be comparable.. so perhaps not.