People love to say "it's early" and "it will improve" about ChatGPT. but amount of training data IS the dominant factor in determining the quality of the output, usually in logarithmic terms. it's already trained on the entire internet. it's hard to see how they'll be able to significantly increase that.
And having models build models is drastically overrated. again, the accuracy/quality improvements are largely driven by the scale and diversity of the dataset. that's like 90% of the solution to any ml problem. choosing the right model and parameters is often a minor relative improvement
And (with scary privacy implications) maybe the next frontier is capturing all spoken language uttered by people in real-time and streaming it into a model that is being updated in near-real-time.
And finally brain implants extracting unspoken thoughts and neural activity and combining it all. Extend it to non human life forms as well. (universal consciousness?)
With feedback.
If there were a model that could adequately replace the role of the human, then that approach would probably work well.
On pure # of words, sure.
But for humans, language is actually just a compressed version of reality perceived through multiple senses / prediction + observation cycles / model paradigms / scales / contexts/ social cues, etc. and we get full access to the entire thing. So a single sentence is wrapped in orders of magnitude more data.
We also get multiple modes of interconnected feedback. How to describe this? Let me use an analogy. In poker, different properties a player has statistically take different amounts of data to reach convergence: Some become evident in 10s of hands, some take 100s of hands, and some take 1000s, and some even take 10,000s before you get over 90% confidence. ....And yet, if you let a good player see your behavior on just one single hand that goes to showdown, a good human player will be able to estimate your playing style, skill, and where your stats will converge to with remarkable accuracy. They get to see how you acted pre-flop, on the flop, turn, and river, with the rich context of position, pot-size, and what the other players were doing during those times, along with the stakes and location you're playing at, what you're wearing, how you move and handle your chips, etc. etc.
But it may be better data.
So, you take your baby AI and instead of training it just on a static corpus, you put it in a simulator. And then when you again hit a wall where you conclude data is the problem, you give them a robot body and plug them into the internet bidirectionally. Some would argue this would be a mistake.
This is mostly wrong.
It's possible to get higher quality results in specific domains by fine-tuning on carefully annotated datasets. However these higher quality results wouldn't be possible without the vast pre-training on the huge dataset LAION-5B provides.
> humans tag stuff for machine learning, and make that tagged data set open? Does such a thing exist?
Yes, there are lots. The size of LAION-5B is the innovation here.
A genetical algorithm was also what I was thinking of. One could devise some kind of symbolic (textual) way to represent a wiring/circuit diagram (graph) and evolve the most efficient "learner" using mutation and cross-breeding (e-sex). The earliest GA I read about used Lisp dicing.
As far as "easiest" AI for humans to work with, "Factor Tables" may be a way:
AI tuning then becomes more like accounting instead of a lab with Doc Brown. Factor Tables are much easier to analyze, debug, and modularize than neural nets.
This is a bit incomplete. It goes both ways. Right in the abstract, it says:
"the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled"
You are probably referring to the statement:
"We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant."
So GPT3 and co would perform better with more data, but only up to a certain point. At that point, you would also need to scale up the model size again to get better performance.
But also, GPT3 and co would perform better when they would be scaled down a bit, when you keep the same training data. That is actually what the Chinchilla paper does. Their Chinchilla model is smaller than GPT3 and others, trained with same amount of data (as Gopher), and performs better.
I don't really see a problem in scaling up models even more. This paper just says that you should also scale up the training tokens equally.
As far as I remember, this paper does not quite address whether you can use the same training data twice. I think all statements assume that every training token is only used once.
TFA though also reminds me of the phenomenon in mathematical proofs where some long winded proof eventually comes up but then over time it becomes simplified as more mathematicians try to optimize it, in much the same way as programmers with technical debt ("make it work, make it right, make it fast"), such as with the four color theorem that was until now computer assisted but it seems there is a non computer assisted proof out [1].
I wonder if the problem in TFA could itself be solved by machine learning, where models would create, train, and test other models, changing them along the way, similar to genetic programming but with "artificial selection" and not "natural selection" so to speak.
[0] https://news.ycombinator.com/item?id=34098087
[1] https://news.ycombinator.com/item?id=34082022