News for hackers. Stuff that matters.
Re:A ChatGPT clone, in 3000 bytes of C, backed by GPT-2 by GTP on Thursday December 12, 2024 @06:44AM (#42398374)
> Could this "data collection" code not potentially be put into a few thousand lines also?

This would mean hardcoding a model of the world. Which maybe, with the help of some breakthrough from physics, would be possibe (but I think the kind of breakthrough needed to get down to such a small size would be a theory of everything). But this means eliminating the self-learning part of current neural networks, which is what makes them attractive: you don't have to hardcode the rules, as the model will be able to learn them.

Re:A ChatGPT clone, in 3000 bytes of C, backed by GPT-2 by n4r9 on Thursday December 12, 2024 @06:52AM (#42398413)
What I mean is, we could program an entity which gathers its own data and updates its own weights. With the lines of code being in the tens of thousands.
Re:A ChatGPT clone, in 3000 bytes of C, backed by GPT-2 by GTP on Thursday December 12, 2024 @07:06AM (#42398482)
But op's point was about people talking about tiny LLMs failing to consider the amount of parameters used. This has nothing to do with the ability of autonomously collecting the training data.