News for hackers. Stuff that matters.

A comment on New York Times considers legal action against OpenAI as copyright tensions swirl. Parent comment

Re:New York Times considers legal action against OpenAI as c... by prmoustache on Thursday August 17, 2023 @03:37AM (#37158430)
ATMO you shouldn't have to maintain knowledge of what kind of crawler bot exist and having to maintain deny list. It should be the opposite, only expressedly allowed content should be crawled by mainaining allow lists.
[ Parent | on HN ]
Re:New York Times considers legal action against OpenAI as c... by wraptile on Thursday August 17, 2023 @04:07AM (#37158599)
You can do the opposite since the inception of robots.txt: User-agent: * Disallow: / and then whitelist google bot and whatnot. Most of the web is already configured this way. Just check robots.txt of any major website, e.g. https://twitter.com/robots.txt
[ Parent | on HN ]
Re:New York Times considers legal action against OpenAI as c... by xnx on Thursday August 17, 2023 @12:34PM (#37164294)
The Allow: directive was an extension to robots.txt added later.
[ Parent | on HN ]