News for hackers. Stuff that matters.

A comment on Google Says It'll Scrape Everything You Post Online for AI. Parent comment

Re:Google Says It'll Scrape Everything You Post Online for AI by tteggel on Tuesday July 04, 2023 @12:07PM (#36588523)
Google does not respect robots.txt if a page is linked to from other sites

https://developers.google.com/search/docs/crawling-indexing/...

[ Parent | on HN ]
Re:Google Says It'll Scrape Everything You Post Online for AI by djmips on Wednesday July 05, 2023 @06:34AM (#36598377)
That doesn't say the content but only that it is indexed.
[ Parent | on HN ]