News for hackers. Stuff that matters.

A comment on Devs say AI crawlers dominate traffic, forcing blocks on entire countries. Parent comment

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by saganus on Tuesday March 25, 2025 @10:20PM (#43478191)
Maybe there is a way for the server to ask the client to do the work?

Something similar to proof-of-work but on a much smaller scale than Bitcoin.

[ Parent | on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by mrheosuper on Tuesday March 25, 2025 @11:51PM (#43478715)
just add some delay to your response, we don't have to waste any more energy on meaningless calculation.
[ Parent | on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by 01HNNWZ0MV43FF on Wednesday March 26, 2025 @01:35AM (#43479161)
Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper
[ Parent | on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by Ma8ee on Wednesday March 26, 2025 @05:09AM (#43480230)
How expensive is it to just keep a connection open?
[ Parent | on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by swiftcoder on Wednesday March 26, 2025 @07:29AM (#43481000)
About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)
[ Parent | on HN ]