News for hackers. Stuff that matters.

A comment on Introduction to web scraping with Python. Parent comment

Re:Introduction to web scraping with Python by anewhnaccount2 on Tuesday October 24, 2017 @08:18AM (#15540990)
Notice how it's not a grey area when Google do it. The usually double standard apply I guess.
[ Parent | on HN ]
Re:Introduction to web scraping with Python by davvolun on Wednesday October 25, 2017 @12:02PM (#15551251)
I don't understand what this comment is referring to. Google's spider respects robots.txt, just block all paths and google will not crawl your site. So too for Bing, Yahoo, Baidu (some complications though, I think), Yandex.... Most of the major spiders respect robots.txt.

Is there some major Google web scraping effort I'm not aware of?

[ Parent | on HN ]