Re:Meta fed its AI on everything adults have publicly posted...
by PaulHoule on Friday September 13, 2024 @02:02PM
(#41533543)
Actually if you want to train a model that can recognize bad things you want to have those bad things in the language model training data, otherwise it won’t see the characteristics of those things and it will later struggle to recognize them in later training stages.
[ Parent
| on HN ]
Re:Meta fed its AI on everything adults have publicly posted...
by btown on Friday September 13, 2024 @02:19PM
(#41533713)
Presence may be necessary, but researcher-driven weighting of different sources of content can still introduce bias. For instance, [0] suggests (sources are unclear) that OpenAI boosted by 5x the weight of their WebText2 dataset, which consists of sites linked to by upvoted Reddit comments. Reddit, in this sense, with all the biases of its various communities, was artificially elevated in importance. (Per [1], there were well-thought-through reasons for this around previous failures due to overreliance on Common Crawl, but it's nonetheless a choice that was made by humans to go in this direction.)
[0] https://gregoreite.com/drilling-down-details-on-the-ai-train...
[1] https://insightcivic.s3.us-east-1.amazonaws.com/language-mod...