AI-Shunning robots.txt

Re:AI-Shunning robots.txt by tbeseda on Thursday March 28, 2024 @11:50AM (#39852989)

Thanks for the work on this!

I automated my site's robots.txt[0] by scraping your site. It would be extra nice if darkvisitor.com exposed a plain text version or JSON representation of the list.

[0] https://tbeseda.com/blog/automating-my-robots-txt-to-block-a...

[ Parent | on HN ]

Re:AI-Shunning robots.txt by glynnormington on Thursday March 28, 2024 @12:00PM (#39853185)

That was one of the points of the new repo. A plain text version of the file is https://raw.githubusercontent.com/ai-robots-txt/ai.robots.tx...

The other point was to make this community maintained rather than rely on one source to provide all the inputs.

[ Parent | on HN ]

Re:AI-Shunning robots.txt by tbeseda on Thursday March 28, 2024 @12:09PM (#39853380)

Definitely! And I'll likely use that raw URL in my tooling going forward. So thanks for starting the repo.

I'm thinking it also helps to bring up a feature request on the source material so we can all limit the drift. ie. if darkvisitors.com had a sort of plain text API, your repo could check for new entries via GH Actions and create issues or even PRs.

[ Parent | on HN ]