Ask HN: Non-Malicious Bots

Posted by andai on Saturday November 07, 2026 @07:18AM
from the ask-hn dept.

When I ask an AI to load a link, sometimes it can load it, and sometimes it can't (and I have to copy paste the text myself).

Sometimes it can load it even when it has the "checking if you're a malicious bot". I'm wondering how that works. Isn't AI scraper traffic literally the thing that page is designed to stop?

Do they let Google, OpenAI etc. user agents through? Or do those companies just have a bunch of guys in a warehouse somewhere with Chrome open, forwarding the AI's HTTP requests?

( 1 comment | on HN )

Re:Ask HN: Non-Malicious Bots by gus_massa on Sunday November 08, 2026 @06:37PM (#50418915)

Most of the times it's just a check that the bot is not making too many request, sometimes using a recent request list, sometimes asking the browser to make a very hard calculation that takes a few seconds to slow down everyone. It does not distinguish good bots and bad bots, just try to protect the server against too many requests.

[ Parent | on HN ]