News for hackers. Stuff that matters.

A comment on Microsoft CEO Testifies That Google’s Power in Search Is Ubiquitous. Parent comment

Re:Microsoft CEO Testifies That Google’s Power in Search Is ... by havnagiggle on Monday October 02, 2023 @08:14PM (#37746261)
Got any reports/statistics to back this up? I highly doubt websites are not wanting major search engines to index them. AFAIK it's been standard practice to use `User-agent: *` for a long time. There are other anti-crawling measures because the bad crawlers are not going to respect your robots.txt.
[ Parent | on HN ]
Re:Microsoft CEO Testifies That Google’s Power in Search Is ... by jwilliams on Tuesday October 03, 2023 @03:58AM (#37749082)
It's not necessarily a lot of sites that block non-popular bots - but often it's big sites (i.e. content-centric sites such as Social Media). Think Yelp, Twitter, LinkedIn, Instagram, etc.

That can add up to a serious percentage of the web.

[ Parent | on HN ]