News for hackers. Stuff that matters.
Re:Ads.txt by organsnyder on Monday March 18, 2024 @01:22PM (#39747183)
Couldn't this be handled in robots.txt?
Re:Ads.txt by omoikane on Monday March 18, 2024 @01:39PM (#39747419)
> Couldn't this be handled in robots.txt?

That would require knowing all the user agents that would scrape content, assuming that you want to only exclude AI scrapers and not search engines in general.

OpenAI's user agent is GPTBot[1], I am not sure about the others.

[1] https://news.ycombinator.com/item?id=37030568

Re:Ads.txt by AnthonyMouse on Monday March 18, 2024 @01:56PM (#39747684)
How are you proposing to distinguish between "AI" and "search engines"? Most of the search engines now have a summarizer at the top which is presumably LLM output, and search engines operate on the basis of ML in general.
Re:Ads.txt by xigoi on Tuesday March 19, 2024 @10:19AM (#39755984)
> Most of the search engines now have a summarizer at the top which is presumably LLM output, and search engines operate on the basis of ML in general.

There is still a difference between scraping content for the purpose of searching it and training on it.

Re:Ads.txt by AnthonyMouse on Wednesday March 20, 2024 @05:07AM (#39764207)
A search engine is an AI model that outputs search results. Creating the index is training it. There is no obvious principled way to distinguish them.
Re:Ads.txt by 8organicbits on Monday March 18, 2024 @01:37PM (#39747398)
Not well. You'd need to know which user-agent strings the AI scrapers use, which is impossible to enumerate.