Incidentally I was about to say that even speculation based on present utility can get a little bit out there. Modern AI is already quite good at generating code snippets, and it already increases developer productivity, but just how much is that going to really change things? Just how much more productive is an AI-augmented software developer? Are LLMs really the thing that was missing from no-code website / app builders?

I wouldn't say it's obvious that this stuff hasn't been scaled to the limit. I mean, sure, you could make bigger LLMs, but I don't think it's a given that LLM performance will increase in performance in a predictable fashion. If anything I think that performance probably resembles an S curve where performance matches improvements in scale before sharply plateauing. When's the plateau?

GPT-4 is a clear improvement over GPT-3, but it's also 6x larger, and so far I haven't quite seen any real evidence to suggest that it's 6x better, other than the fact that it scored better on tests.

It's hard to say what "6x better" even means, but there are tasks that GPT-3.5 consistently fails hard at that GPT-4 is much better at (and has improved over time).

I think the next big leap will be the context size. Going from 4k to 32k tokens means that the model can see a lot more when generating the answer. For code especially this makes a big difference in the ability to reason about it.

As to the limits, it's hard to predict because what we're seeing is more of a series of sudden leaps. But let me put it this way - so far we've seen noticeable improvements on every iteration of the new LLMs. So given this, why should the default assumption be that the limit past which nothing new will emerge, whatever it is, is close to the present size of the publicly available models?

Is anyone doing static code analysis with GPT? It seems like an excellent idea. I think a combination of formal analysis methods together with blackbox GPT magic will be a killer combination. That said, the market is tiny.

There is zero evidence that GPT-4 is a trillion parameter model and some evidence that it is the same parameter size as GPT-3.

Newer research and models out of DeepMind etc. show that GPT-4 performance is attainable at 90% the size of GPT-3. It's well known by now that GPT-3 was hugely over parameterized.

I think it’s extremely dangerous to predict the pattern of gains to continue.

Also I’m not actually convinced that there’s much value provided currently. I’ve yet to see a particularly compelling use case, and every time I go to try something that folks are hyping up (image generation, ChatGPT, copilot) I leave feeling like it’s all sizzle and no steak.

It's probably more dangerous to predict they won't, at this point.

The people that aren't seeing a compelling use case simply aren't looking.

> The people that aren't seeing a compelling use case simply aren't looking.

By all means, go do your startup, all the power to you.

But I personally fail to see it creating enough value for people to even bother use the AI, even if it's free and any problem could be ignored.

I can imagine people adopting it if it comes embedded on whatever software they already use. But that doesn't "completely change everything" or any of the other things people are repeating.

And yeah, I can't imagine it quickly improving so that happens either. I am personally bracing for a new AI winter, and believe that word is going to become more toxic than it has ever been.