Reddit has updated its robots.txt to block all web crawlers

Re:Reddit has updated its robots.txt to block all web crawlers by Anarch157a on Thursday July 04, 2024 @08:34AM (#40874400)

The article says Google is paying Reddit to get the data directly from their firehose API, so they wont even bother crawling the public website.

[ Parent | on HN ]

Re:Reddit has updated its robots.txt to block all web crawlers by The_Colonel on Thursday July 04, 2024 @09:14AM (#40874691)

I wonder how much they pay. Reddit profits a lot from showing up on the top for many search queries. I very often do "whatever I'm looking for reddit" (for e.g. product reviews), since the reddit results often provide higher quality information than normal results.

[ Parent | on HN ]

Re:Reddit has updated its robots.txt to block all web crawlers by toomuchtodo on Thursday July 04, 2024 @11:08AM (#40875519)

$60M/year

https://www.reuters.com/technology/reddit-ai-content-licensi...

[ Parent | on HN ]

Re:Reddit has updated its robots.txt to block all web crawlers by rbetts on Thursday July 04, 2024 @08:59AM (#40874580)

I wonder if these indexing deals will become more antitrust evidence.

[ Parent | on HN ]