News for hackers. Stuff that matters.

A comment on We should start to add “ai.txt” as we do for “robots.txt”. Parent comment

Re:We should start to add “ai.txt” as we do for “robots.txt” by asdkjlfkasjdf on Wednesday May 10, 2023 @07:33PM (#35895061)
exactly, but nobody in their sane minds would be using robots.txt to block crawlers as that is SEO (organic traffic from searches) disaster.

People could be adding a specific robot user-agent, if they knew openai even existed before yesterday and was stealing their content. but nobody did.

[ Parent | on HN ]

There are no comments on this comment yet.