Re:Amazon spends another $2.7B on Anthropic
by benreesman on Wednesday March 27, 2024 @03:48PM
(#39843729)
I’m on record as saying that we need to squeeze the water out of these bloated models.
It’s possible to scale better than N^2 for some value of “better”. OpenAI has yet to demonstrate that they have the elusive combination of technical sophistication and institutional health to do so. Mistral can run a better model as judged by outcomes on my Mac Studio than GPT-4 is on an Azure disagg rack. Altman seems to understand this.
I’m on record as saying he’s amoral, non-technical, and a clear and present danger to Enlightenment civilization, not that he’s stupid.
I think his math on what it’s going to cost an OpenAI that Karpathy wants nothing to do with to reach the next level is refreshingly candid.
Note that scaling up model size doesn't necessarily refer to context size anyway - may also be things like embedding dimensions size, number of transformer layers, number of experts (MoE), etc.
In general scaling up 10x in size and/or cost between generations is about as much as makes sense, and about as much as can be achieved in a year. Anthropic have explicitly talked about $1B models coming soon, so this isn't just speculation.