Re:Reddit has updated its robots.txt to block all web crawlers
by contrarian1234 on Thursday July 04, 2024 @07:22AM
(#40873996)
That's probably the whole point ..
I'd say I add +reddit to a third of my searches these days
Now I'll have to go to their shitty built-in search and they can algorithmically feed me garbage and make lots of money from people that pay them
Re:Reddit has updated its robots.txt to block all web crawlers
by littlecranky67 on Thursday July 04, 2024 @07:24AM
(#40874000)
Smart move actually. The "+reddit" is something more and more people do on google, and on kagi.com reddit results are ranken No. 1 usually. Seems Googles search result quality will drop even further now.
EDIT: Article explains there is a 60M deal from Google using reddits API so that they can continue delivering results from Reddit. Will only hurt smaller search engines, like Kagi :(
Re:Reddit has updated its robots.txt to block all web crawlers
by Expurple on Thursday July 04, 2024 @08:26AM
(#40874346)
Kagi in particular shouldn't be affected. Apart from using its own index, it also aggregates results from other search engines, most likely including Google. https://help.kagi.com/kagi/search-details/search-sources.htm...
> Disallow: /
Ugh oh, that means all search engines are gona delist reddit content.