In it's current form, no chance. But if we continue on this trajectory (if that's possible of course), then pretty soon. I personally think there will be a barrier popping up soon for further advancement. However, today I am picking olives (not a joke) for my new career if it should happen (a joke; I would just retire).

You don't think it's already plateaued? In terms of reasoning ability, anyway.

Well yes, that's the feeling I have, but there are now so many very smart people working on smarter ways and billions are poured in (not only on the 'AI for car searches' type of sites...); I'm willing to give it some benefit of the doubt.

Not even close. You can look at the Llama v2 training curves and its clear that even 7B models are still under-trained and they themselves have only been trained on 2T tokens. A 70B or whatever size/model amalgamation GPT-4 is also likely under-trained.

Apart from just plain under-training, there's that whole 'grokking' phenomenon that's been observed in smaller models, where it looks like they're not really improving for a long time, but _eventually_ suddenly undergo a massive improvement. I don't know if anyone has been willing to set enough cash on fire to see if grokking can happen in a large LLM and/or how long it would need to be trained for.

There's also still a lot of juice left in improving the quality of datasets and in training on larger data sets. OpenAI, having built GPT-4, probably want to work on some applications of it given that they have enough of a lead on everyone else. I think there's definitely a few "burn piles of cash to train a better model" buttons at OpenAI right now, but they have no reason to use them when they're clearly in the lead anyway.

There's also the actual limiting factor: there are only so many A100s/H100s in the world, but the amount is growing.

Yeah, even RL tuning can be applied to improve reasoning paths for specialized applications, something like RLAIF by anthropic but for specialized agents

using cursor as my editor, its close as of today, I mostly steer the AI into directions rather than typing on my own majority of time now. Especially complex refactorings or debugging/error hunting works surprisingly well when the full codebase can be used as embedded vectors on demand automatically

Give it 1 year

Yeah, you kind of want to say, "GPT, turn the cube into a sphere," and not have to do the clicks/drags yourself.

Yeah, need a UI overlay assist atleast. sort of like hints in games