News for hackers. Stuff that matters.
Re:Ask HN: Any way to ban GPT-4 and others from harvesting m... by tareqak on Sunday August 14, 2022 @02:44AM (#32456995)
That is a good point in the case the blogger still wants their blog to be indexed and found in search results, but does not want their content to be used as ML training data.

However, I still think that a program that crawls the web to collect data should respect robots.txt irrespective of what it does with the collected data both during the crawl and afterwards.

I would strongly assume that someone who doesn’t want their website to be indexed and found in search results would also likely not want their content to be processed as ML training data since it too reads the content, but I might be wrong here.

There are no comments on this comment yet.