News for hackers. Stuff that matters.

A comment on The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch. Parent comment

Re:The web does not need gatekeepers: Cloudflare’s new “sign... by jeroenhd on Saturday August 30, 2025 @03:35AM (#45072684)
I have had to disable at least one service because AI bots kept hitting it and it started impacting other stuff I was running that I am more interested in. Part of it was the CPU load on the database rendering dozens of 404s per second (which still required a database call), part of it was that the thumbnail images were being queried over and over again with seemingly different parameters for no reason.

I'm sure there are AI bots that are good and respect the websites they operate on. Most of them don't seem to, and I don't care enough about the AI bubble to support them.

When AI companies stop people from using them as cheap scrapers, I'll rethink my position. So far, there's no way to distinguish any good AI bot from a bad one.

[ Parent | on HN ]
Re:The web does not need gatekeepers: Cloudflare’s new “sign... by account42 on Monday September 01, 2025 @06:18AM (#45091347)
> Part of it was the CPU load on the database rendering dozens of 404s per second (which still required a database call)

That's one request every 80 ms which is an eternity in CPU time. How the hell can you not afford to check that something doesn't exist every 80 ms.

> part of it was that the thumbnail images were being queried over and over again with seemingly different parameters for no reason.

Is there a reason you are serving thumbnails for arbitrary parameters?

[ Parent | on HN ]