News for hackers. Stuff that matters.

A comment on Turn entire websites into LLM-ready data. Parent comment

Re:Turn entire websites into LLM-ready data by donohoe on Saturday April 13, 2024 @08:16PM (#40027497)
And you honor or ignore robots.txt?
[ Parent | on HN ]
Re:Turn entire websites into LLM-ready data by cpeffer on Saturday April 13, 2024 @08:27PM (#40027557)
It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
[ Parent | on HN ]