The hype exists, firstly, because the present day possibilities are already impressive enough to net actual financial gains from improved productivity, and secondly because it's obvious that all this stuff hasn't yet been scaled to the limit - but the scaling and improvement that we've seen so far (GPT 3 -> 3.5 -> 4) showed massive gains on every step in a very short period of time.

Incidentally I was about to say that even speculation based on present utility can get a little bit out there. Modern AI is already quite good at generating code snippets, and it already increases developer productivity, but just how much is that going to really change things? Just how much more productive is an AI-augmented software developer? Are LLMs really the thing that was missing from no-code website / app builders?

I wouldn't say it's obvious that this stuff hasn't been scaled to the limit. I mean, sure, you could make bigger LLMs, but I don't think it's a given that LLM performance will increase in performance in a predictable fashion. If anything I think that performance probably resembles an S curve where performance matches improvements in scale before sharply plateauing. When's the plateau?

GPT-4 is a clear improvement over GPT-3, but it's also 6x larger, and so far I haven't quite seen any real evidence to suggest that it's 6x better, other than the fact that it scored better on tests.

It's hard to say what "6x better" even means, but there are tasks that GPT-3.5 consistently fails hard at that GPT-4 is much better at (and has improved over time).

I think the next big leap will be the context size. Going from 4k to 32k tokens means that the model can see a lot more when generating the answer. For code especially this makes a big difference in the ability to reason about it.

As to the limits, it's hard to predict because what we're seeing is more of a series of sudden leaps. But let me put it this way - so far we've seen noticeable improvements on every iteration of the new LLMs. So given this, why should the default assumption be that the limit past which nothing new will emerge, whatever it is, is close to the present size of the publicly available models?

Is anyone doing static code analysis with GPT? It seems like an excellent idea. I think a combination of formal analysis methods together with blackbox GPT magic will be a killer combination. That said, the market is tiny.

There is zero evidence that GPT-4 is a trillion parameter model and some evidence that it is the same parameter size as GPT-3.

Newer research and models out of DeepMind etc. show that GPT-4 performance is attainable at 90% the size of GPT-3. It's well known by now that GPT-3 was hugely over parameterized.

I think it’s extremely dangerous to predict the pattern of gains to continue.

Also I’m not actually convinced that there’s much value provided currently. I’ve yet to see a particularly compelling use case, and every time I go to try something that folks are hyping up (image generation, ChatGPT, copilot) I leave feeling like it’s all sizzle and no steak.

It's probably more dangerous to predict they won't, at this point.

The people that aren't seeing a compelling use case simply aren't looking.

> The people that aren't seeing a compelling use case simply aren't looking.

By all means, go do your startup, all the power to you.

But I personally fail to see it creating enough value for people to even bother use the AI, even if it's free and any problem could be ignored.

I can imagine people adopting it if it comes embedded on whatever software they already use. But that doesn't "completely change everything" or any of the other things people are repeating.

And yeah, I can't imagine it quickly improving so that happens either. I am personally bracing for a new AI winter, and believe that word is going to become more toxic than it has ever been.

I think generative AI has incremental utility. You can't half-use Bitcoin or the Metaverse; the idea is nonsensical. Bitcoin moves data around very slowly and at enormous cost, and the Metaverse provides vertigo-inducing VRChat for uncanny valley Mii avatars, but they're only usable if you go all-in.

Even if generative AI is not a gamechanging buzzword paradigm shift it's still useful for making existing tasks easier. Predicting the next sentence is autocomplete for word documents and phone keyboards, and we're already seeing it roll out. Artists who've gotten over the AI backlash can use it for backgrounds or textures, and designers can churn out those stylised low detail corporate-website graphics. Amateur authors on sites like RoyalRoad are already creating cover art and character portraits with AI generators. Millions of horny internet weirdos now have access to oddly specific pornography, and even if the AI gets the hands wrong they can retouch it in Krita.

Probably (>50%) we'll all be bored of it in 20 years and admit the hype was overblown, but I also think we'll see it integrated into day-to-day tasks in a way that Bitcoin and the Metaverse aren't.

That's kind of what I was betting on. I don't see it going away anytime soon, but I also think it's potential impact is being overblown.

With regards to AI image generators, I think people overestimate the importance of illustration skills to visual creatives. Namely, if you look at the top graphic designers, something you pick up on is that they aren't necessarily known for their top-tier illustration skills. Anyone can theoretically make a logo in the style of Paul Rand, yet curiously, hardly anyone ever does.

Even if the person in question is an illustrator, I think what fundamentally makes them valuable as an illustrator is that they know what to draw rather than how to draw.

It's sentiments like these that make me question the notion of certain careers getting disrupted by AI as opposed to simply improved by it.