Re:EU to unveil landmark law curbing power of tech giants
by dmitriid on Tuesday December 15, 2020 @11:59AM
(#25432125)
Google's crawler gets preferential treatment on many sites. Not necessarily because of private deals, but because it drives traffic from search results.
Quote via [2]
--- start quote ---
When Mr. Maril started researching how sites treated Google’s crawler, he downloaded 17 million so-called robots.txt files — essentially rules of the road posted by nearly every website laying out where crawlers can go — and found many examples where Google had greater access than competitors.
ScienceDirect, a site for peer-reviewed papers, permits only Google’s crawler to have access to links containing PDF documents. Only Google’s computers get access to listings on PBS Kids. On Alibaba.com, the U.S. site of the Chinese e-commerce giant Alibaba, only Google’s crawler is given access to pages that list products.
Re:EU to unveil landmark law curbing power of tech giants
by sib on Tuesday December 15, 2020 @03:56PM
(#25435145)
So it sounds like the regulators should focus on forcing every web property to give equal access to all crawlers. Why steal Google's investment and hard work and innovation?
Quote via [2]
--- start quote ---
When Mr. Maril started researching how sites treated Google’s crawler, he downloaded 17 million so-called robots.txt files — essentially rules of the road posted by nearly every website laying out where crawlers can go — and found many examples where Google had greater access than competitors.
ScienceDirect, a site for peer-reviewed papers, permits only Google’s crawler to have access to links containing PDF documents. Only Google’s computers get access to listings on PBS Kids. On Alibaba.com, the U.S. site of the Chinese e-commerce giant Alibaba, only Google’s crawler is given access to pages that list products.
--- end quote ---
[1] https://www.nytimes.com/2020/12/14/technology/how-google-dom...
[2] https://daringfireball.net/linked/2020/12/14/wakabayashi-goo...