News for hackers. Stuff that matters.
Re:LLMs Are This Close to Destroying the Internet by Kamq on Sunday April 14, 2024 @12:02PM (#40032078)
That seems like it would mean you're restricted to a dataset from before 2022ish or you have to use data that's significantly more expensive to clean.

That seems like it might make the training process untenable.

Re:LLMs Are This Close to Destroying the Internet by coffeebeqn on Sunday April 14, 2024 @12:37PM (#40032354)
Is it really true though? If they scrape HN today I don’t think their sampling would be that “polluted” compared to 2022.

If you were to teach a LLM to play chess today - wouldn’t you use fully AI generated data because that’s by far the best data ?

Re:LLMs Are This Close to Destroying the Internet by johntb86 on Sunday April 14, 2024 @03:40PM (#40033769)
Isn't all of recorded human history up until 2022 enough to train a pretty smart ai? It'll miss out on future trends, but some targeted training on trusted news sources should be enough. Eventually languages will change enough that the old training data will seem dated, but that's going to take a while to happen.
Re:LLMs Are This Close to Destroying the Internet by Kamq on Sunday April 14, 2024 @05:44PM (#40034750)
Yeah, but basically all of that comes from post 2000.

Information production seems to be on an exponential curve, so it won't be that long before you're training on a minority of all data.