On a similar note, I once found a paper that explained artificial neural networks computability from a very tiny ground up method, showing how few pieces you need to build a NAND gate... and of course once you have that you have everything.
What really stood out to me is that, while the typical wisdom is “more data -> better model”, in the video they use the exact same data for all the models, so all the gains are achieved only through improved algorithms and computation power
With that in mind, I wonder if at some point someone will figure out an algorithm for a model that can be trained on just an English dictionary and get a decent LLM-equivalent model that can have a basic conversation. Given the small size of the data (a dictionary), I assume the model would be pretty small and quite fast to run, even on older or smaller machines
Interestingly, this meant that the conventional wisdom that the test and train distributions have to be identical for optimal held-out performance, is not true at small model sizes. It is true as models grow larger - and I was explicitly able to show this. See [2] - where the x-axis is model size, and the y-axis measures (on a scale of 0-1) how close is the optimal training distribution to the test distribution. The different lines are for different datasets.
These images are from the paper here [3]. I have a library too [4]; that is in need of updates - it works today as-is though, but please use the latest minor release if this of interest [5]!
[1] https://imgur.com/a/NheK49Z
[2] https://imgur.com/a/N53GeNI
[3] https://arxiv.org/pdf/1906.06852.pdf
[4] https://compactem.readthedocs.io
[5] As of writing the install command should be `pip install compactem==0.9.9rc2`
I've got some geometric handwaving for why this works, as well. It's easier to find a low-energy solution when you have more parameters... Sparse solutions are higher energy, and thus require longer walks (and more commitment) during training.
I'm thinking it would help reduce the network demands for gradient updates if merge from time to time. That could unlock distributed training, like SETI@Home.
The drop and rescale method outlined in the paper makes the latent space increasingly sparse, which in turn allows weights to merged without much interference or degradation.
My instinct is that while merging models will have some use cases, ultimately these insights will lead to innovations in training and architecture that have the same result but with better computational efficiency.
For example instead of training an 8x7b mixture of experts then merging, just incorporate the sparsity constraint while pre-training a single 7b model (somehow).