News for hackers. Stuff that matters.

A comment on AI bots everywhere. Does anyone have a good whitelist for robots.txt?. Parent comment

Re:AI bots everywhere. Does anyone have a good whitelist for... by scoofy on Tuesday January 28, 2025 @11:26PM (#42861521)
I’m fine with a burst of traffic… that’s what it’s designed for. What it’s not designed for is 24-7-365 being slowly pinged on every single page (thousands), by every single robot (thousands).
[ Parent | on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for... by dapperdrake on Wednesday January 29, 2025 @12:35AM (#42861911)
<snark> Maybe the bots are hallucinating off-by-one errors and then 25-8-367 isn’t all that bad. </snark>
[ Parent | on HN ]