Given how they've struggled to even make even GPT 4 economical it's highly unlikely that larger models would be in any way cost effective for wide use.

And there's certainly more to be found in training longer on better datasets and adjusting the architecture.

It's important to remember that "not cost-effective" is not the same as "not useful", though.

Depends on how you look at it I guess. If it's really too expensive and overly slow to run for end users but is able to say, generate perfect synthetic datasets over time which can then be used to train smaller and faster models which pay back for that original cost then it is cost effective, just in a different application.

What I mean, rather, is that "cost-effective" implies that everything can be reduced down to some costing. So e.g. having humans do the mind-numbing work is often more "cost-effective" than robots doing the same thing, because humans can be hired for less. But from a broader perspective, a robot replacing a human in such a role is socially beneficial even if the robot is more expensive - and the ideal state of affairs is when all things that people would rather not do are fully automated away. LLMs are one piece of that puzzle, but we can only put it together if we stop thinking about everything in terms of money saved.

In capitalism everything can and is reduced down to some costing indeed. We don't have public schools and art classes because it's nice, it's because an educated society as a whole has a much higher economic output and you need someone to design good looking products which will sell better. I wouldn't say I agree with the process, but you can't ignore that it's the underlying reason for almost everything our society does.

On topic of LLMs and diffusion models I doubt they're really having that much of a positive social impact, I would bet that in the long run they'll automate more of the jobs humans would like to do compared to the ones we don't, and erode trust on a global level. But the economic gain is undeniable so we're doing it anyway. Automating all work would be a good thing in an ideal world, but in practice it's the one thing that still gives the lower classes some leverage over the capitalists and a world where the average person is expendable and unemployable isn't good for society.

I feel like this is a self-defeating approach. We don't need an ideal world for that - you rightly identified capitalism as the problem, and it's not utopian to think of other options. Nor do many of them require some sort of ideal man to work, as their opponents sometimes present.

Change will happen - necessarily so, in fact, as more automation is brought online, because the crowd of people excluded from the economy as "redundant" will not be happy about that state of affairs, and it will only grow in size. At some point, they will understand that the real problem is not that they don't have work, but that they don't have a fair share of the automation pie. And things like, say, property rights on means of production / automation stop to matter if most people don't believe in them anymore.

I don't think that's what that means. I think he means "we don't have to keep making models larger, because we've found ways to make very good models that are also small".