Re:We should start to add “ai.txt” as we do for “robots.txt”
by jefftk on Wednesday May 10, 2023 @09:38AM
(#35887183)
> It's not a system that we should copy for preventing the use of content in training AI
I don't see the OP saying anything about "ai.txt" being for that? They're advocating it as a way that AIs could use fewer tokens to understand what a site is about.
(Which I also don't think is a good idea, since we already have lots of ways of including structured metadata in pages, but the main problem is not that crawlers would ignore it.)
Re:We should start to add “ai.txt” as we do for “robots.txt”
by kmoser on Wednesday May 10, 2023 @11:57AM
(#35889221)
Not only do we already have lots of ways of including structured metadata, but if you want to include directives about what should/shouldn't be scraped and by whom, we already have robots.txt.
In other words, there's no need to create an ai.txt when the robots.txt standard can just be extended.
Re:We should start to add “ai.txt” as we do for “robots.txt”
by Wowfunhappy on Wednesday May 10, 2023 @09:28AM
(#35887045)
> But my point is that it's not a system that we should copy for preventing the use of content in training AI.
I actually don't think that's what OP is envisioning based on their post.
Re:We should start to add “ai.txt” as we do for “robots.txt”
by capableweb on Wednesday May 10, 2023 @09:31AM
(#35887096)
> But my point is that it's not a system that we should copy for preventing the use of content in training AI
The purpose OP is suggesting in the submission is the opposite, help AI crawlers to understand what the page/website is about without actually having to infer the purpose from the content itself.
Re:We should start to add “ai.txt” as we do for “robots.txt”
by Xelynega on Wednesday May 10, 2023 @09:51AM
(#35887351)
Isn't that the entire point of the semantic web?
Re:We should start to add “ai.txt” as we do for “robots.txt”
by kmoser on Wednesday May 10, 2023 @11:59AM
(#35889249)
If only there was an HTML tag that let you provide a concise description of the page content. Perhaps something like <meta name="description" content="This is an example of a meta description. This will often show up in search results.">
Re:We should start to add “ai.txt” as we do for “robots.txt”
by BurningFrog on Wednesday May 10, 2023 @09:30AM
(#35887087)
OP is trying to give helpful info to the AI, not set boundaries for it.
If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static).
With an ai.txt there is no potential downside to violating it - unless we get new legislation enforcing its legal standing. The nature of ML models is that it's opaque what content exactly it's trained on, there is no obvious retaliation or retribution.