News for hackers. Stuff that matters.

A comment on Reddit has updated its robots.txt to block all web crawlers. Parent comment

Re:Reddit has updated its robots.txt to block all web crawlers by littlecranky67 on Thursday July 04, 2024 @07:24AM (#40874000)
Smart move actually. The "+reddit" is something more and more people do on google, and on kagi.com reddit results are ranken No. 1 usually. Seems Googles search result quality will drop even further now.

EDIT: Article explains there is a 60M deal from Google using reddits API so that they can continue delivering results from Reddit. Will only hurt smaller search engines, like Kagi :(

[ Parent | on HN ]
Re:Reddit has updated its robots.txt to block all web crawlers by Expurple on Thursday July 04, 2024 @08:26AM (#40874346)
Kagi in particular shouldn't be affected. Apart from using its own index, it also aggregates results from other search engines, most likely including Google. https://help.kagi.com/kagi/search-details/search-sources.htm...
[ Parent | on HN ]