News for hackers. Stuff that matters.
Re:We should start to add “ai.txt” as we do for “robots.txt” by brookst on Wednesday May 10, 2023 @09:10AM (#35886826)
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.

Failing to solve every problem does not mean a solution is a failure.

From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.

Re:We should start to add “ai.txt” as we do for “robots.txt” by samwillis on Wednesday May 10, 2023 @09:16AM (#35886902)
Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction.

If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static).

With an ai.txt there is no potential downside to violating it - unless we get new legislation enforcing its legal standing. The nature of ML models is that it's opaque what content exactly it's trained on, there is no obvious retaliation or retribution.

Re:We should start to add “ai.txt” as we do for “robots.txt” by jefftk on Wednesday May 10, 2023 @09:38AM (#35887183)
> It's not a system that we should copy for preventing the use of content in training AI

I don't see the OP saying anything about "ai.txt" being for that? They're advocating it as a way that AIs could use fewer tokens to understand what a site is about.

(Which I also don't think is a good idea, since we already have lots of ways of including structured metadata in pages, but the main problem is not that crawlers would ignore it.)

Re:We should start to add “ai.txt” as we do for “robots.txt” by kmoser on Wednesday May 10, 2023 @11:57AM (#35889221)
Not only do we already have lots of ways of including structured metadata, but if you want to include directives about what should/shouldn't be scraped and by whom, we already have robots.txt.

In other words, there's no need to create an ai.txt when the robots.txt standard can just be extended.

Re:We should start to add “ai.txt” as we do for “robots.txt” by Wowfunhappy on Wednesday May 10, 2023 @09:28AM (#35887045)
> But my point is that it's not a system that we should copy for preventing the use of content in training AI.

I actually don't think that's what OP is envisioning based on their post.

Re:We should start to add “ai.txt” as we do for “robots.txt” by capableweb on Wednesday May 10, 2023 @09:31AM (#35887096)
> But my point is that it's not a system that we should copy for preventing the use of content in training AI

The purpose OP is suggesting in the submission is the opposite, help AI crawlers to understand what the page/website is about without actually having to infer the purpose from the content itself.

Re:We should start to add “ai.txt” as we do for “robots.txt” by Xelynega on Wednesday May 10, 2023 @09:51AM (#35887351)
Isn't that the entire point of the semantic web?
Re:We should start to add “ai.txt” as we do for “robots.txt” by kmoser on Wednesday May 10, 2023 @11:59AM (#35889249)
If only there was an HTML tag that let you provide a concise description of the page content. Perhaps something like <meta name="description" content="This is an example of a meta description. This will often show up in search results.">
Re:We should start to add “ai.txt” as we do for “robots.txt” by BurningFrog on Wednesday May 10, 2023 @09:30AM (#35887087)
OP is trying to give helpful info to the AI, not set boundaries for it.
Re:We should start to add “ai.txt” as we do for “robots.txt” by bilekas on Wednesday May 10, 2023 @09:50AM (#35887344)
> Failing to solve every problem does not mean a solution is a failure.

There is something to be said though to OP's point where it's actually better to do nothing than an AI.txt because it can give a false sense of security, which is obviously not what you want.

Re:We should start to add “ai.txt” as we do for “robots.txt” by lelanthran on Wednesday May 10, 2023 @12:53PM (#35890017)
The point of an ai.txt is that it signals intention of the copyright holder.

Anytime a business is caught using that content, they can't claim that they used publicly available information, because the ai.txt specifically signalled to everyone in a clear and unambiguous manner that the copyright granted by viewing the page is witheld from ai training.

Re:We should start to add “ai.txt” as we do for “robots.txt” by usrusr on Wednesday May 10, 2023 @09:57AM (#35887442)
That's still not an argument to introduce ai.txt, because everything a hypothetical ai.txt could ever do is already done just as good (or not) by the robots.txt we have. If a training data crawler ignores robots.txt it won't bother checking for an ai.txt either.

And if you feel like rolling out the "welcome friend!" doormat to a particular training data crawler, you are free to dedicate as detailed a robots.txt block as you like to its user agent header of choice. No new conventions needed, everything is already on place.

Re:We should start to add “ai.txt” as we do for “robots.txt” by michaelmior on Wednesday May 10, 2023 @11:08AM (#35888483)
This seems to be assuming a very different purpose for ai.txt than the OP proposed. It sounds like they are intending ai.txt to give useful contextual information to crawlers collecting AI training data. Robots.txt does not have any of this information (although I suppose you could include it in comments).
Re:We should start to add “ai.txt” as we do for “robots.txt” by irobeth on Wednesday May 10, 2023 @11:24AM (#35888750)
worse, ai.txt could become an adversarial vector for attempts to trick the AI into filing your information under some semantic concept
Re:We should start to add “ai.txt” as we do for “robots.txt” by joshuanapoli on Wednesday May 10, 2023 @04:11PM (#35892807)
I do think that robots.txt is pretty useful. If I want my content indexed, I can help the engine find my content. If indexing my content is counterproductive, then I can ask that it be skipped. So it helps the align my interests with the search engine; I can expose my content or I can help the engine avoid wasting resources indexing something that I don't want it to see.

It would also be useful to distinguish training crawlers from indexing crawlers. Maybe I'm publishing personal content. It's useful for me to have it indexed for search, but I don't want an AI to be able to simulate me or my style.

Re:We should start to add “ai.txt” as we do for “robots.txt” by vlunkr on Wednesday May 10, 2023 @10:53AM (#35888254)
I know it's getting pedantic, but sunscreen and seatbelts are a poor analogy. They do offer protection if you use them. robots.txt only offers protection if other people/robots choose to care about them.