News for hackers. Stuff that matters.

A comment on Llms.txt. Parent comment

Re:Llms.txt by itsbjoern on Wednesday September 04, 2024 @12:01AM (#41441770)
Always find it amusing when people write about „blocking“ requests using robots.txt as if they are deploying a firewall
[ Parent | on HN ]
Re:Llms.txt by gitgud on Wednesday September 04, 2024 @05:58AM (#41443808)
Agreed, all it takes is another site to copy the content, then an LLM could just scrape that…

An open web that block scraping… is likely “not an open web”

[ Parent | on HN ]