I think the next big leap will be the context size. Going from 4k to 32k tokens means that the model can see a lot more when generating the answer. For code especially this makes a big difference in the ability to reason about it.
As to the limits, it's hard to predict because what we're seeing is more of a series of sudden leaps. But let me put it this way - so far we've seen noticeable improvements on every iteration of the new LLMs. So given this, why should the default assumption be that the limit past which nothing new will emerge, whatever it is, is close to the present size of the publicly available models?
Newer research and models out of DeepMind etc. show that GPT-4 performance is attainable at 90% the size of GPT-3. It's well known by now that GPT-3 was hugely over parameterized.
I wouldn't say it's obvious that this stuff hasn't been scaled to the limit. I mean, sure, you could make bigger LLMs, but I don't think it's a given that LLM performance will increase in performance in a predictable fashion. If anything I think that performance probably resembles an S curve where performance matches improvements in scale before sharply plateauing. When's the plateau?
GPT-4 is a clear improvement over GPT-3, but it's also 6x larger, and so far I haven't quite seen any real evidence to suggest that it's 6x better, other than the fact that it scored better on tests.