Stop crawling my HTML – use the API

Re:Stop crawling my HTML – use the API by frogperson on Sunday December 14, 2025 @02:40PM (#46266077)

We need a crowd sourced list like adgaurd, but for bots. Id love to block all those ips at the firewall.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by mrweasel on Sunday December 14, 2025 @04:15PM (#46266995)

So that would be at least: GCP, Azure, Alibaba, AWS, Huawei, AT&T, BT, Cox... it's a long list.

User Agents then? No, because that would be: Chrome and Safari.

It's an uphill battle, because the bot authors do not give a shit. You can now buy bot network from actual companies, who embed proxies in free phone games. Anthropic was caught hiding behind Browserbase, and neither of the companies seems to see problem with that.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by jarofgreen on Sunday December 14, 2025 @03:48PM (#46266737)

User agents not IPs, but: https://github.com/ai-robots-txt/ai.robots.txt

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by dotancohen on Sunday December 14, 2025 @03:01PM (#46266277)

A large portion of those addresses will be valid residential IP addresses running malware on compromised Windows machines.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by venturecruelty on Sunday December 14, 2025 @03:04PM (#46266314)

Block GCP, AWS, Azure, and various datacenter prefixen, and you're pretty much golden. There are scant few legitimate reasons a human being's traffic would originate from those hosts.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by bdcravens on Sunday December 14, 2025 @03:20PM (#46266449)

You can run virtual desktops in the cloud, like AWS's Workspaces, sold as a business rather than developer offering. AWS does publish the IP range those clients use, and I assume other similar offerings out there do the same.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by johneth on Sunday December 14, 2025 @03:39PM (#46266649)

I'm sure people who can afford to run virtual desktops in the cloud can also afford a phone/laptop/desktop to access sites that block those virtual desktops in the cloud.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by bdcravens on Sunday December 14, 2025 @05:13PM (#46267612)

I'm thinking more along the lines of people using virtual desktops assigned by their job, and those sites are part of their work. I don't feel like punting to BYOD is a good solution.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by prmoustache on Sunday December 14, 2025 @04:34PM (#46267184)

I am working from a cloud desktop but I am only visiting corporate approved resources from that cloud desktop and I believe that is the case of most cloud desktop users as the whole point is to have a clear separation of duties.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by bdcravens on Sunday December 14, 2025 @05:17PM (#46267658)

Correct, but I don't think it's a safe assumption that approved resources wouldn't have a reason to block requests from the cloud.

[ Parent | on HN ]

Re:Stop crawling my HTML – use the API by alexspring on Monday December 15, 2025 @10:10AM (#46275480)

The only way you can block these "AI" scrapers is a combination of IP filtering (https://spur.us/) and Fingerprinting (https://abrahamjuliot.github.io/creepjs/).

Things like browserbase are easy to block with this. It's a losing battle though, personally moved entirely to real environments for https://browser.cash/developers

[ Parent | on HN ]