News for hackers. Stuff that matters.

A comment on We should start to add “ai.txt” as we do for “robots.txt”. Parent comment

Re:We should start to add “ai.txt” as we do for “robots.txt” by shanebellone on Wednesday May 10, 2023 @09:01AM (#35886711)
"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare."

I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.

[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by alwayslikethis on Wednesday May 10, 2023 @09:08AM (#35886798)
Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts
[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by ape4 on Wednesday May 10, 2023 @09:41AM (#35887222)
I like this idea. Of course it would have to be only to robots that visit a page disallowed by the robots.txt
[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by shanebellone on Wednesday May 10, 2023 @09:28AM (#35887048)
"Why not serve fake garbage indistinguishable from real content by a computer, like LLM output?"

Serving more than the minimum wastes resources. Worse yet, a better solution would cost my time.

"Sending errors just incentivizes bot owners to fix the identifiable parts"

Sure, someone could make or configure their scraper perfectly. "Perfect" is now the table stakes though.

Edit:

My solution strives to cause an unproportional expense in order to circumvent. I want 10x on my time.

[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by twelve40 on Wednesday May 10, 2023 @09:31AM (#35887091)
it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.
[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by shanebellone on Wednesday May 10, 2023 @09:36AM (#35887160)
You can. I can't think of the appropriate term though. Hopefully someone else chimes in here with a link.
[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by dspillett on Wednesday May 10, 2023 @09:55AM (#35887400)
> Sending errors just incentivizes bot owners to fix the identifiable parts

Nah. It'll just make them fake their identity so it is harder to tell the traffic is from a bot.

[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by spc476 on Wednesday May 10, 2023 @01:47PM (#35890777)
It might be a better idea to serve up a 418 ("I'm a tea pot") with a line line text file saying "I'm not an HTTP server". That solved a problem I had with bots making HTTP requests to my gopher server [1]. Serving up a 503 informs the bot that there's a server issue and it may try again later. A 418 informs the bot that it made an erroneous request and such an odd error code might get someone to look into it and stop.

[1] https://boston.conman.org/2019/09/30.2

[ Parent | on HN ]
Re:We should start to add “ai.txt” as we do for “robots.txt” by shanebellone on Friday May 12, 2023 @09:01AM (#35915603)
This is very interesting. I've bookmarked the link. Thanks for sharing. I believe minimal is best and this might fit nicely within my larger system. Do you approach other problems with a similar mindset?
[ Parent | on HN ]