News for hackers. Stuff that matters.
Re:Reddit has updated its robots.txt to block all web crawlers by littlecranky67 on Thursday July 04, 2024 @07:18AM (#40873979)
> User-agent: *

> Disallow: /

Ugh oh, that means all search engines are gona delist reddit content.

Re:Reddit has updated its robots.txt to block all web crawlers by contrarian1234 on Thursday July 04, 2024 @07:22AM (#40873996)
That's probably the whole point ..

I'd say I add +reddit to a third of my searches these days

Now I'll have to go to their shitty built-in search and they can algorithmically feed me garbage and make lots of money from people that pay them

Re:Reddit has updated its robots.txt to block all web crawlers by littlecranky67 on Thursday July 04, 2024 @07:24AM (#40874000)
Smart move actually. The "+reddit" is something more and more people do on google, and on kagi.com reddit results are ranken No. 1 usually. Seems Googles search result quality will drop even further now.

EDIT: Article explains there is a 60M deal from Google using reddits API so that they can continue delivering results from Reddit. Will only hurt smaller search engines, like Kagi :(

Re:Reddit has updated its robots.txt to block all web crawlers by Expurple on Thursday July 04, 2024 @08:26AM (#40874346)
Kagi in particular shouldn't be affected. Apart from using its own index, it also aggregates results from other search engines, most likely including Google. https://help.kagi.com/kagi/search-details/search-sources.htm...