I look back over the past 2-3 years and am pretty amazed with how quick change and progress have been made. The promises are indeed large but the speed of progress has been fast. Not defending the promise but “taking a very long time” does not seem to be an accurate representation.

I guess it probably depends on what you are doing. Outside of layers on top of these things (tooling), I personally haven't seen much progress.

What a time we live in. I guess it depends how pessimistic you are.

To their point, there hasn’t been any huge breakthrough in this field since the “attention is all you need” paper. Not really any major improvements to model architecture, as far as I am aware. (Admittedly, this is a new field of study to me.) I believe one hope is to develop better methods for self-supervised learning; I am not sure of the progress there. Most practical improvements have been on the hardware and tooling side (GPUs and, e.g., pytorch).

Don’t get me wrong: the current models are already powerful and useful. However, there is still a lot of reason to remain skeptical of an imminent explosion in intelligence from these models.

You’re totally right that there hasn’t been a fundamental architectural leap like “attention is all you need”, that was a generational shift. But I’d argue that what we’ve seen since is a compounding of scale, optimization, and integration that’s changed the practical capabilities quite dramatically, even if it doesn’t look flashy in an academic sense. The models are qualitatively different at the frontier, more steerable, more multimodal, and increasingly able to reason across context. It might not feel like a revolution on paper, but the impact in real-world workflows is adding up quickly. Perhaps all of that can be put in the bucket of “tooling” but from my perspective there has still been quite large leaps looking at cost differences alone.

For some reason my pessimism meter goes off when I see single sentence arguments “change has been slow”. Thanks for brining the conversation back.

I'm all for flashy in academic sense, because we can let engineers sort out the practical aspects, especially by combining flashy academic approach. The flaw from LLM architecture can be predicted from the original paper, no amount of engineering can compensate that.

Probably depends on how immature your knowledge of tge subject matter is.

Feel free to share resources, but I am speaking purely in terms of practicality related to my day to day.

> The promises are indeed large but the speed of progress has been fast

And at the same time, absurdly slow? ChatGPT is almost 3 years old and pretty much AI has still no positive economic impact.

Saying “AI has no economic impact” ignores reality. The financials of major players clearly show otherwise—both B2C and B2B applications are already profitable and proven. While APIs are still more experimental, and it’s unclear how much value businesses can ultimately extract from them, to claim there’s no economic impact is willful blindness. AGI may be far off, but companies are already figuring out value from both the consumer side and slowly API.

The financials are all inflated by perception of future impact. This includes the current subscriptions as businesses are attempting to use AI to some economic benefit, but it's not all going to work out to be useful.

It will take some time for whatever reality is to actually show truthfully in the financials. When VC money stops subsidising datacentre costs, and businesses have to weigh the full price against real value provided, that is when we will see the reality of the situation.

I am content to be wrong either way, but my personal prediction is if model competence slows down around now, businesses will not be replacing humans en-mass, and the value provided will be notable but not world changing like expected.

OpenAI alone is on track to generate as much revenue as Asus or US Steel this year ($10-$15 billion). I don't know how you can say AI has had no positive economic impact.

And what is their burn rate? Everyone fails to mention the amount they are spending for this return.

That is not even 1 month of a big tech revenue, it is a global negligible impact. 3 years talking about AI changing the world, 10bi revenue and no ecosystem around making money besides friends and VCs pumping and dumping LLM wrappers.

There's a pretty wide gulf between being one of the most important companies in the global marketplace as Microsoft, Apple, and Amazon are and "having no economic impact".

I agree that most of the AI companies describe themselves and their products in hyperbolic terms. But that doesn't mean we need to counter that with equally absurd opposing hyperbole.

There is no hyperbole. I think AI will change the world in the next 10 years but comparing to the iphone, for example, 3 years the economic impact was much, much bigger and that is just one brand of smartphones.

Revenue, not profit.

If it costs them even just one more dollar than that revenue number to provide that service (spoiler, it does), then you could say AI has had no positive economic impact.

Considering we know they’re being subsidized by obscene amounts of investment money just like all other frontier model providers, it seems pretty clear it’s still a negative economic impact, regardless of the revenue number.

There is the huge blind spot where tech workers think LLMs are being made primarily to either assist them or replace them.

Nobody seems to consider that LLMs are democratizing programming, and allowing regular people to build programs that make their work more efficient. I can tell you that at my old school manufacturing company, where we have no programmers and no tech workers, LLMs have been a boon for creating automation to bridge gaps and even to forgo paid software solutions.

This is where the change LLMs will bring will come from. Not from helping an expert dev write boilerplate 30% faster.

Low code/no code/visual programming has been around forever. They all had issues. LLMs will also have the same issues and cost even more.

I'm not aware of any that you speak/type plain English to.

You never heard of COBOL? It's original premise was you can now use something resembling English to write programs.

I feel like we've made barely any progress. It's still good at the things Chat GPT was originally good at, and bad at the things it was bad at. There's some small incremental refinement but it doesn't really represent a qualitative jump like Chat GPT was originally. I don't see AI replacing actual humans without another step jump like that.

As a non-programmer non-software engineer, the programs I can write with modern SOTA models are at least 5x larger than the ones GPT-4 could make.

LLMs are like bumpers on bowling lanes. Pro bowlers don't get much utility from them. Total noobs are getting more and more strikes as these "smart" bumpers get better and better at guiding their ball.

> I look back over the past 2-3 years and am pretty amazed with how quick change and progress have been made.

Now look at the past year specifically, and only at the models themselves, and you'll quickly realize that there's been very little real progress recently. Claude 3.5 Sonnet was released 11 months ago and the current SOTA models are only marginally better in terms of pure performance in real world tasks.

The tooling around them has clearly improved a lot, and neat tricks such as reasoning have been introduced to help models tackle more complex problems, but the underlying transformer architecture is already being pushed to its limits and it shows.

Unless some new revolutionary architecture shows up out of nowhere and sets a new standard, I firmly believe that we'll be stuck at the current junior level for a while, regardless of how much Altman & co. insist that AGI is just two more weeks away.

Third AI Winter from overpromise/underdeliver when?

Third? It’ll be the tenth or so.