News for hackers. Stuff that matters.
Re:The web does not need gatekeepers: Cloudflare’s new “sign... by danudey on Friday August 29, 2025 @05:50PM (#45069823)
Not only are they scraping without sending traffic, they're doing so much more aggressively than Google ever did; Google, at least, respected robots.txt and kept to the same user-agent. They didn't want to index something that a server didn't want indexed. AI bots, on the other hand, want to index every possible thing regardless of what anyone else says.

There are no comments on this comment yet.