News for hackers. Stuff that matters.

A comment on Devs say AI crawlers dominate traffic, forcing blocks on entire countries. Parent comment

Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by 01HNNWZ0MV43FF on Wednesday March 26, 2025 @01:35AM (#43479161)
Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper
[ Parent | on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by Ma8ee on Wednesday March 26, 2025 @05:09AM (#43480230)
How expensive is it to just keep a connection open?
[ Parent | on HN ]
Re:Devs say AI crawlers dominate traffic, forcing blocks on ... by swiftcoder on Wednesday March 26, 2025 @07:29AM (#43481000)
About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)
[ Parent | on HN ]