Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by kevin_thibedeau on Tuesday March 25, 2025 @10:14PM
(#43478160)
The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.
[ Parent
| on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by tedunangst on Tuesday March 25, 2025 @10:25PM
(#43478228)
Tons of problems are easier to verify than to solve.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by fsckboy on Tuesday March 25, 2025 @11:01PM
(#43478432)
do any of them involve mining bitcoin?
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by Retr0id on Tuesday March 25, 2025 @11:03PM
(#43478442)
sure
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by saganus on Tuesday March 25, 2025 @10:20PM
(#43478191)
Maybe there is a way for the server to ask the client to do the work?
Something similar to proof-of-work but on a much smaller scale than Bitcoin.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by mrheosuper on Tuesday March 25, 2025 @11:51PM
(#43478715)
just add some delay to your response, we don't have to waste any more energy on meaningless calculation.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by 01HNNWZ0MV43FF on Wednesday March 26, 2025 @01:35AM
(#43479161)
Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by Ma8ee on Wednesday March 26, 2025 @05:09AM
(#43480230)
How expensive is it to just keep a connection open?
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by swiftcoder on Wednesday March 26, 2025 @07:29AM
(#43481000)
About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by prisenco on Tuesday March 25, 2025 @10:22PM
(#43478207)
Right it would need an algorithm with widely different encryption speeds vs decryption speeds. Lattice-based cryptography maybe?
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by Retr0id on Tuesday March 25, 2025 @11:06PM
(#43478470)
Hash functions are all you need.
Re:Devs say AI crawlers dominate traffic, forcing blocks on ...
by Dylan16807 on Wednesday March 26, 2025 @01:06AM
(#43479046)
Yeah, searching for hashes with some prefix is easy to set up.