Devs say AI crawlers dominate traffic, forcing blocks on entire countries

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by karlgkk on Wednesday March 26, 2025 @12:05AM (#43478777)

One thing that worked well for me was layering obstacles

It really sucks that this is the way things are, but what I did was

10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count

After a captcha pass, 100 requests in an hour gets you auth walled

It’s really shitty but my industry is used to content scraping.

This allows legit users to get what they need. Although my users maybe don’t need prolonged access ahem.

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by nomel on Wednesday March 26, 2025 @12:20AM (#43478851)

What happens if you use the proper rate limiting status of 429? It includes a next retry time [1]. I'm curious what (probably small) fraction would respect it.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by karlgkk on Friday March 28, 2025 @02:41AM (#43502237)

Probably makes sense for a b2b app where you publish status codes as part of the api

Bad actors don’t care and annoying actors would make fun of you for it on twitter

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by LPisGood on Wednesday March 26, 2025 @12:32AM (#43478901)

What is your website?

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by rfurmani on Wednesday March 26, 2025 @12:40AM (#43478937)

I've wanted to but wasn't sure how to keep track of individuals. What works for you? IP Addresses, cookies, something else?

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by karlgkk on Friday March 28, 2025 @02:43AM (#43502260)

I use IP addy. Users behind cgnat are already used to getting captcha the first time around

There’s some stuff you can do, like creating risk scores (if a user changes ip and uses the same captcha token, increase score). Many vendors do that, as does my captcha provider.

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by nukem222 on Wednesday March 26, 2025 @01:59AM (#43479246)

> This allows legit users to get what they need.

Of course they could have just used the site directly.

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by karlgkk on Friday March 28, 2025 @02:41AM (#43502247)

If bots and scrapers respected the robots and tos, we wouldn’t be here

It sucks!

[ Parent | on HN ]

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by GoblinSlayer on Wednesday March 26, 2025 @06:02PM (#43487935)

Or just buy cloudflare :)

[ Parent | on HN ]