News for hackers. Stuff that matters.

A comment on Reddit has updated its robots.txt to block all web crawlers. Parent comment

Re:Reddit has updated its robots.txt to block all web crawlers by schroeding on Thursday July 04, 2024 @08:24AM (#40874321)
It can be ignored (it's the equivalent to a "keep out" sign on a lawn), but I very much doubt Google et al. (Edit: Oops, Bing et al.) will actually ignore it.
[ Parent | on HN ]
Re:Reddit has updated its robots.txt to block all web crawlers by Anarch157a on Thursday July 04, 2024 @08:34AM (#40874400)
The article says Google is paying Reddit to get the data directly from their firehose API, so they wont even bother crawling the public website.
[ Parent | on HN ]
Re:Reddit has updated its robots.txt to block all web crawlers by The_Colonel on Thursday July 04, 2024 @09:14AM (#40874691)
I wonder how much they pay. Reddit profits a lot from showing up on the top for many search queries. I very often do "whatever I'm looking for reddit" (for e.g. product reviews), since the reddit results often provide higher quality information than normal results.
[ Parent | on HN ]
Re:Reddit has updated its robots.txt to block all web crawlers by toomuchtodo on Thursday July 04, 2024 @11:08AM (#40875519)
$60M/year

https://www.reuters.com/technology/reddit-ai-content-licensi...

[ Parent | on HN ]
Re:Reddit has updated its robots.txt to block all web crawlers by rbetts on Thursday July 04, 2024 @08:59AM (#40874580)
I wonder if these indexing deals will become more antitrust evidence.
[ Parent | on HN ]
Re:Reddit has updated its robots.txt to block all web crawlers by dzhiurgis on Thursday July 04, 2024 @05:42PM (#40878299)
Google sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)
[ Parent | on HN ]