If I remember correctly, their counter-intuitive result was that big overparameterized models could learn more efficiently, and were less likely to get trapped in poor regions of the optimization space.
[This is also similar to how introducing multimodal training gives an escape hatch to get out of tricky regions.]
So with this hand-wavey argument, it might be the case that two-phase training is needed: A large overcomplete pretraining focused on assimilating all the knowledge, and a second that makes it compact. Other, that there is a hyperparameter that controls overcompleteness vs compactness and you adjust it over training.
Vs trying to fill something with just a narrow tube, you spill most of what you put in.
This is a mischaracterization of sparsity. Performance did drop, so the weights are not gibberish. Training vs pruning, you can't train into the final state, you can only prune there.
Going forward we will accumulate truly useful data at a linear growing rate. This fundamentally breaks the scaling game. If your model and compute expand exponentially but your training data only linearly, the efficiency won't be the same.
Synthetic data might help us pad up the training sets, but the most promising avenue I think is to use user-LLM chat logs. Those logs contain real world grounding and human in the loop. Millions of humans doing novel tasks. But that only scales linearly with time, as well.
No way around it - we only once had the whole internet for the first time in the training set. After that it's linear time.
I have a copy of refined web locally so I have a billion pre-chatgpt documents for my long term use.
> Also, most of them are silent. They don’t really do much. Or their activities are… You have to hit it with just the right set of stimulus.
> ... When you place these electrodes, again, within this hundred micron volume, you have 40 or so neurons. Why do you not see 40 neurons? Why do you see only a handful? What is happening there?
(Yes, I understand LLM aren't brains.)
My intiution tells me the pre-training paradigm will shift immensely in near future because we started to understand that we don’t need all these paramaters since the subnetworks seems to be very robust preserving information in high dimensions. We keep saying curse of dimensionality but it is more like the bliss of dimensionality we keep seeing. Network redundancy still seems to be very high given BitNet is more less comparable to other LLMs.
This basically shows over 50% of the neural net is gibberish! The reason being is that the objective function simply does not include it.
Again my intiution tells me that neural scaling laws are incomplete as they are because they lack the efficiency parameter that needs to be taken into account (or simply left out due to greed of corporate).
And this is what we are seeing as “the wall”.
I am no expert in neural network theory nor in math but I would assume the laws should be something in the vicinity of this formulation/simulation:
https://colab.research.google.com/drive/1xkTMU2v1I-EHFAjoS86...
and encapsulate shannon’s channel’s capacity. I call them generalized scaling laws since it includes what it should include in the first place: entropy.