News for hackers. Stuff that matters.

A comment on The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch. Parent comment

Re:The web does not need gatekeepers: Cloudflare’s new “sign... by Symbiote on Friday August 29, 2025 @06:27PM (#45070106)
Google (and the others) crawl from a published IP range, with "Google" in the user agent. They read robots.txt. They are very easy to block

The AI scum companies crawl from infected botnet IPs, with the user agent the same as the latest Chrome or Safari.

[ Parent | on HN ]
Re:The web does not need gatekeepers: Cloudflare’s new “sign... by davepeck on Friday August 29, 2025 @07:16PM (#45070475)
Okay. Which, specifically, are the "AI scum" companies you're speaking of?

There are plenty of non-AI companies that also use dubiously sourced IPs and hide behind fake User-Agents.

[ Parent | on HN ]
Re:The web does not need gatekeepers: Cloudflare’s new “sign... by Symbiote on Saturday August 30, 2025 @05:20AM (#45073210)
I don't know which companies, of course. They hide their identity by using a botnet.

This traffic is new, and started around when many AI startups started.

I see traffic from new search engines and other crawlers, but it generally respects robots.txt and identifies itself, or else comes from a small pool of IP addresses.

[ Parent | on HN ]
Re:The web does not need gatekeepers: Cloudflare’s new “sign... by lostmsu on Saturday August 30, 2025 @09:08AM (#45074321)
Why do you think the bots you see are AI scum companies?
[ Parent | on HN ]