News for hackers. Stuff that matters.

A comment on How can I prevent my site from being a free dataset for LLMs?.

Re:How can I prevent my site from being a free dataset for L... by vhcr on Tuesday April 11, 2023 @01:17AM (#35521131)
The training on GPT was done on Common Crawl, Reddit, books, and Wikipedia.

For Common Crawl, the documentation says blocking it on robots.txt should work, as for Wikipedia, Reddit, and books, there's no option than to not participate AFAIK.

OpenWebText2 has no mention of robots.txt, so good luck with that.

[ Parent | on HN ]

There are no comments on this comment yet.