I've heard conflicting things and I'm curious to hear your reasoning!
See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682
Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728
All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).
Loss= (N0/N)^a+(D0/D)^b
Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.
Loss = A/parameters + B/dataset_size + C
Just read the chinchilla paper.
What more corpus is even available? If the corpus is a matter of human annotation, what exactly is being annotated in say the Illiad? A tweet that says, "You up?". The lyrics to 8 Mile. So on and so forth.
And the frontier after is probably having a fleet of actual robots capture additional information, as per this paper.
The performance does scale up with parameters, though it’s not linear.
As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget.
So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.