News for hackers. Stuff that matters.
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by og_kalu on Tuesday May 16, 2023 @05:11AM (#35959336)
>Sam Altman) have already publicly acknowledged that the diminishing returns they're seeing in terms of model size are sufficient to effectively declare that 'the age of giant models is already over.'

He never said anything about technical diminishing returns. He's saying we're hitting a wall economically.

The Chief Scientist at Open AI thinks there's plenty of ability left to squeeze out.

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by somenameforme on Tuesday May 16, 2023 @10:25AM (#35962331)
You can see his comments, in context, here: https://youtu.be/T5cPoNwO7II?t=356

Economics was not hinted or implied in any way. Diminishing returns on model size doesn't mean there's nothing left to squeeze out, it just means that what gains are made are going to be in model refinement, rather than going the NVidia vision of a quadrillion weight system and expecting large, or even linear, gains from that hop up in model size.