Yes and no.

The performance does scale up with parameters, though it’s not linear.

As discovered by Google in their work on their Chinchilla LLM, performance also scales up with the size of your training set. They were able to do work to define the optimal amount of training for a model of a given size to get the most out of your budget.

So even if we don’t find any better model architectures, which we probably will, if we increase the size of our models, training corpus and budget, we should continue to get more performant models.

How does that performance scale though? If you double the inputs you train on, do you get twice the performance? Less? More?

I've heard conflicting things and I'm curious to hear your reasoning!

The scaling appears linear in relationship, though on some performance metrics it is super linear.

See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682

Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728

All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has basically become a meme now for how much he poopoos LLMs and insists what we're seeing are LLMs only pretending to reason).

Chinchilla paper proposes the following empirical scaling power law

Loss= (N0/N)^a+(D0/D)^b

Where N is number of parameters, D is number of examples, a,b are exponents. N0,D0 are scaling factors.

Where A,B,C are empirical constants,

Loss = A/parameters + B/dataset_size + C

Just read the chinchilla paper.

I asked ChatGPT, it said: "Uhhh it depends."

The corpus part is the one that gets me. Presumably Google, Facebook, Microsoft, etc have a mirror of essentially the entire public (and their private) internet communications. An enormous amount of published English literature has been digitized.

What more corpus is even available? If the corpus is a matter of human annotation, what exactly is being annotated in say the Illiad? A tweet that says, "You up?". The lyrics to 8 Mile. So on and so forth.

I would assume that video and audio are the next frontier. Talking about Google, training on the entirety of YouTube would probably be next, if they haven't yet used it.

And the frontier after is probably having a fleet of actual robots capture additional information, as per this paper.

From what I know, parameter count correlates directly with capability to a certain degree. But Google has also shown there is a benefit in training that can enable lower parameter count models to match or even outperform those with bigger counts.

The trade off is in the cost for training. But in return, you get a smaller and cheaper to run model that can be used by more people.

That is not what PALM E is about though. This is the paper where they patch an LLM so that it can give command to a real, physical robot and basically let the LLM interact with meatspace.

Suddenly those jailbreaks of the LLM become a lot more serious. What happens when bad people can get a robot to help them commit various crimes?

What happens when it decides that the human interfering in the example is the source of the problem and needs to be eliminated?

All I could think of watching that video was I would not want to be that guy.

As far as I’m aware the only “robots” we have at the moment that are capable of large scale physical damage, without direct human involvement while the damage is occurring, are military drones and missiles.

And neither missiles nor drones are capable of functioning totally independent with both requiring large amounts of weapon systems infrastructure. You may be able to argue missiles could be deployed almost independently if failsafes are circumvented but that would require modernization of missile weapon systems which is about as likely as the colonization of space.

At present I’d be more worried about network and social engineering attacks using LLM which I’m sure is already happening and will continuously increase.

The household robot that can cook will be able to use a knife to cut anything it arbitrarily thinks needs to be cut which could lead to it being directed to assist with serious criminal acts. Self-driving cars out on the streets have already killed people, but those were just bugs, not intentional behavior, as might occur if they were driven by LLMs.

Then the bad person gets charged with the crimes. It's not like a judge would just throw their hands in the air because there is a small amount of indirection.

Parameters and training corpus size matter. Last year a new paper (Google "Chinchilla optimality") focused on compute-optimal LLMs found that we'd been under-training models -- i.e. you could wring more performance out of smaller models by training on more data. But (as far as I understand it - interested layman here) - for a given amount of data, model performance seems to scale more or less linearly with parameters.

Now, we could see another model architecture than the current reigning transformer architecture upend this (much work is ongoing on breaking the quadratic term in the transformer that computationally bounds its performance - an example is the Hyena paper that was published just the other day).

Biggest computer and most data wins is still the paradigm here.

For now increasing data size has been enough to show better results. We don't when diminishing returns will be significant.

Kind of.

Speaking very generally, it's not just the number of weights ("parameters") but how you use them. The architecture of the neural network is where the most interesting new work is taking place.

Stable Diffusion, which was beating out Dall-E2, was a big deal not just because the weights were public, but also because they were small enough to run on consumer hardware. This is because of the clever architectural choice (plus some other methods.) This was surprising to me, because most of the interesting image generation required huge models running on commercial GPUs.

You'd generally expect diminishing returns from adding weights (as well as a lot more training time and maybe more data) but this is not a law.

  I know I could just ask chatgpt but wanted to support human content on the internet.

The irony of this sentence. As if the millionth repetitive copy of defeatist doomsday commentary on chatGPT is somehow a very different kind of content

Does it still count as human content if I blindly copy paste from ChatGPT? :))

That last phrase hit me hard.

ChatGPT is human content on the internet. It's just indexed and compressed and was made "generative" in a mind boggling way.