News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by supert56 on Friday March 26, 2021 @01:00PM (#26594364)
Perhaps I am misunderstanding or over simplifying things but it always surprises me that there are legal cases brought against companies who scrape data when so many of Google's products are doing exactly this.

It definitely feels like one set of rules for them and a different set for everyone else.

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by Spivak on Friday March 26, 2021 @02:00PM (#26595026)
I mean it's not that weird that a company would authorize major search engines scraping them but no one else.

I don't really see this as Google playing by different rules so much as economic incentives being aligned in Google's favor.

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by lupire on Friday March 26, 2021 @05:53PM (#26597426)
Google doesn't scrape anything that the site owner objects to.
[ Parent | on HN ]