News for hackers. Stuff that matters.

A comment on AI-Shunning robots.txt.

Re:AI-Shunning robots.txt by internetter on Thursday March 28, 2024 @08:46AM (#39850572)
This is missing a couple, one that comes to mind is `FriendlyCrawler`, which is most definitely not friendly, and very likely for AI
[ Parent | on HN ]
Re:AI-Shunning robots.txt by glynnormington on Thursday March 28, 2024 @12:02PM (#39853223)
Feel free to submit a PR. :-)
[ Parent | on HN ]