You can't possibly know that, given that we don't actually understand how LLMs work on a high level.
> We've pretty much tapped out that depth with GPT4
GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?
It is possible they've reached some 80/20 point and he is pretty honest about how much more extendable the current approach really is.
Would explain going to congress and asking for regulation (of their not-quite-there-yet competitors who they want a regulatory moat against).
It's a fair assumption to make however - basically 80/20 rule.
AI research isn't a new thing and I bet you could go back 40/50 years where they thought they were about to have a massive breakthrough to human level intelligence.
> GPT-4 is three months old and you're confident that its working principle cannot be extended further? Where do you get that confidence from?
I'm guessing from actually using it.
GPT4 is super impressive and helpful in a practical way, but having used it myself for a while now I get this feeling also. It feels a bit like "it's been fed everything we have, with all the techniques we have, now what?"
It's going to take time to figure out what works and what doesn't.
There's a reason why Sam Altman is saying they're not training GPT5, and it's not because they think GPT4 is good enough.
Are you saying that people who created ChatGPT don't understand how it works? Or that we the rest of people don't?
I'd wager it's far more likely 5 years than 50 LLMs get to the full depths all humans are capable of. Simply compare the state of LLMs today vs 2018.
I'd say this is immediately counterindicated by the available evidence. Gpt2 was hopeless for anything other than some fun languagw games like a bot replica of a subreddit or trump. 3.5 is much much bigger, and has semi competent but limited reasoning abilities.
Gpt 4 is a vast improvement over 3.5 in various reasoning tasks. Yes, a priori I would have agreed with you that this has to stop somewhere, but not anymore. I would need to see some data of post gpt4 models to believe you.
If you really poke at GPT, you begin to realize it's fairly shallow. Human intelligence is like a deep well or pond, where as GPT is a vast but shallow ocean.
Making that ocean deeper is not a trivial problem that we can just throw more compute or data at. We've pretty much tapped out that depth with GPT4 and are going to need better designs.
This could only take half a decade or it could be half a century. Plenty of enterprises stagnate for decades.