News for hackers. Stuff that matters.

A comment on AI bots everywhere. Does anyone have a good whitelist for robots.txt?.

Re:AI bots everywhere. Does anyone have a good whitelist for... by _blk on Tuesday January 28, 2025 @11:16PM (#42861448)
Takes balls to put the URL in a HN post if you're looking to reduce traffic costs.. Of course it's legit traffic for once but I assume curiosity just got the better half of them.. .. On the constructive side: shield it with cloudflare.
[ Parent | on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for... by scoofy on Tuesday January 28, 2025 @11:26PM (#42861521)
I’m fine with a burst of traffic… that’s what it’s designed for. What it’s not designed for is 24-7-365 being slowly pinged on every single page (thousands), by every single robot (thousands).
[ Parent | on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for... by dapperdrake on Wednesday January 29, 2025 @12:35AM (#42861911)
<snark> Maybe the bots are hallucinating off-by-one errors and then 25-8-367 isn’t all that bad. </snark>
[ Parent | on HN ]