News for hackers. Stuff that matters.

A comment on Google is a monopoly. Breakup may be coming; what comes after may not be better. Parent comment

Re:Google is a monopoly. Breakup may be coming; what comes a... by macNchz on Thursday August 15, 2024 @09:10AM (#41255571)
Websites these days are often pretty hostile to crawling aside from the Googlebot. Other crawlers are asked not to by the robots.txt, fully blocked, or put through captchas.

https://knuckleheads.club/

https://www.nytimes.com/2020/12/14/technology/how-google-dom...

[ Parent | on HN ]
Re:Google is a monopoly. Breakup may be coming; what comes a... by jsheard on Thursday August 15, 2024 @09:27AM (#41255791)
Good point. Now I think about it Google is actively contributing to that with their Reddit deal which has them blocking all scrapers besides Googlebot, even known legit search engine crawlers like Bingbot.
[ Parent | on HN ]