Not sure where you are getting quadratic from... Increasing context size had quadratic cost using original attention, but it seems everyone has switched to newer more efficient attention schemes. Claude-3 is being experimented with 10x context size of GPT-4 (1M vs 128K), but surely didn't cost 10^2 x $100M to train!

Note that scaling up model size doesn't necessarily refer to context size anyway - may also be things like embedding dimensions size, number of transformer layers, number of experts (MoE), etc.

In general scaling up 10x in size and/or cost between generations is about as much as makes sense, and about as much as can be achieved in a year. Anthropic have explicitly talked about $1B models coming soon, so this isn't just speculation.

I’m on record as saying that we need to squeeze the water out of these bloated models.

It’s possible to scale better than N^2 for some value of “better”. OpenAI has yet to demonstrate that they have the elusive combination of technical sophistication and institutional health to do so. Mistral can run a better model as judged by outcomes on my Mac Studio than GPT-4 is on an Azure disagg rack. Altman seems to understand this.

I’m on record as saying he’s amoral, non-technical, and a clear and present danger to Enlightenment civilization, not that he’s stupid.

I think his math on what it’s going to cost an OpenAI that Karpathy wants nothing to do with to reach the next level is refreshingly candid.