Re:StarCoder and StarCoderBase: 15.5B parameter models with ...
by somenameforme on Tuesday May 16, 2023 @10:25AM
(#35962331)
You can see his comments, in context, here: https://youtu.be/T5cPoNwO7II?t=356
Economics was not hinted or implied in any way. Diminishing returns on model size doesn't mean there's nothing left to squeeze out, it just means that what gains are made are going to be in model refinement, rather than going the NVidia vision of a quadrillion weight system and expecting large, or even linear, gains from that hop up in model size.
He never said anything about technical diminishing returns. He's saying we're hitting a wall economically.
The Chief Scientist at Open AI thinks there's plenty of ability left to squeeze out.