News for hackers. Stuff that matters.
Re:Amazon vs. Perplexity – U.S. Court of Appeals for the Nin... by shermantanktop on Monday September 14, 2026 @08:28PM (#49706131)
> robots.txt is a service to crawlers to help them avoid wasting time

I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping crawlers be efficient.

Re:Amazon vs. Perplexity – U.S. Court of Appeals for the Nin... by Zambyte on Monday September 14, 2026 @10:49PM (#49707087)
Well that's what they're doing.