Re:Ask HN: Any way to ban GPT-4 and others from harvesting m...
by deltree7 on Sunday August 14, 2022 @02:16AM
(#32456859)
That also means his blog won't be indexed and won't be found in search results
[ Parent
| on HN ]
Re:Ask HN: Any way to ban GPT-4 and others from harvesting m...
by tareqak on Sunday August 14, 2022 @02:44AM
(#32456995)
That is a good point in the case the blogger still wants their blog to be indexed and found in search results, but does not want their content to be used as ML training data.
However, I still think that a program that crawls the web to collect data should respect robots.txt irrespective of what it does with the collected data both during the crawl and afterwards.
I would strongly assume that someone who doesn’t want their website to be indexed and found in search results would also likely not want their content to be processed as ML training data since it too reads the content, but I might be wrong here.
Re:Ask HN: Any way to ban GPT-4 and others from harvesting m...
by capableweb on Sunday August 14, 2022 @12:19PM
(#32460452)
You can create a robots.txt that blocks everything but known bots (by name, like googlebot).