I keep hearing this but the current evidence, asymptotic progress and financials say otherwise.

I guess what you are saying would probably have been said by AI skeptics in the 70s, but LLMs provided a quantum leap. Yes, progress is often asymptotic and governed by diminishing returns, but discontinuous breakthrough must also be factored in.

Please tell me what quantum leap was provided by LLMs. Please inform me of any developments that made current LLMs possible.

I contend that there are none. Witness the actual transformer kernel technologies over the last 20 years and try to find a single new one.

Neural Networks? that's 90's technology. Scale is the only new factor I can think of.

This is an investor-dollar driven attempt to use brute-force to deliver "magical" results when the fundamental technology is being mis-represented to the general public, to CTOs, and even to Developers.

This is dishonest and will not end well.

Nailed it.

The biggest capability jump comes from semantic search. You can now search based on the content of a text rather than a literal character level match.

I am slowly coming 'round to the same conclusion: Word2Vec might be as fundamental as fire - all due caveats aside, of course ...

Technological advances tend to happen in jumps. We are no doubt approaching a local optima right now, but it won't be long until another major advancement propels things forward again. We've seen the same pattern in ML for decades.

Please name me one technological advance of major import in the fundamental transformer kernel space that has occurred in the last decade that has any import at all on today's LLMs.

I will wait.

The very idea of the Transformer architecture. Surely you've heard of "Attention is all you need".

Ah, yes, AI jam tomorrow.

(Though, perhaps an unusually pessimistic example of the “real soon now, it’ll be usable, we promise” phenomenon; rifles took about 250 years to go from ‘curiosity’ to ‘somewhat useful’).

it is unlikely that the output of LLMs will improve. there is no fundamental breakthrough in transformer technology (or anything else) powering todays' LLM "revolution"

There is only scale being employed like never before => vast datasets being plowed through being sufficient to provide the current illusion for the less observant humans out there...

10 years from now this current fad of LLM's pretending to be intelligent will look preposterous and unbelievable: "how COULD they all have fallen for such hype, and what a cost of joules/computation... the least deterministic means possible of coming to any result... just wasteful for no purpose..."

On the other hand, firearms were invented in the 10th century, and I don't think we got reliable cartridges until mid 1800's.

All that time the bow and arrow were long-range, accurate, quiet and worked in the rain. :)

[But yeah, you're right - in our lifetimes technologies have changed immensely.]