Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.

The runtime is quadratic for a given context size, although it seems like there is some progress on this front https://gwern.net/note/attention

Exponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query.

Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.

Compute will also scale exponentially in coming years. The data source limitation seems to be a harder barrier, I think many companies are experimenting with AI generated content for training at this point.

> Compute will also scale exponentially in coming years.

Cost per transistor scaling has already plateaued or perhaps even inverted with TSMC's latest and greatest.

And the new chips, even after 25 layers of EUV lithography, more than doubling the previous record, and an extra year of fine tuning, has total SRAM size scaling of -5% and logic scaling of -42%.

These are numbers verified by experienced semi people.

INSUFFICIENT DATA FOR MEANINGFUL ANSWER

https://users.ece.cmu.edu/~gamvrosi/thelastq.html

I wasnt aware that it was his favorite story, thanks for posting it

Yes - image generation AIs also take longer to generate higher resolution pictures.

I'd rather have a LLM that thinks a bit longer than a LLM that spits out wrong answers immediately.

Idk. I’m not the one making the claim we’ve already nailed the optimal point though.