Predicting the next token well is indistinguishable from intelligence. The bubble is financial, as you point out, just like railroads, radio and dotcom bubbles were. The technology didn’t die, only the companies did. LLMs are in the exploit phase, that doesn’t mean there isn’t more to explore and there may be an architecture-hardware combination that takes us an order of magnitude or two higher in a step-function-like fashion just like LLMs did 2024-2026.

> entire premise on whether it is a bubble depends on if LLMs can be more than a next token prediction machine or not

I don't think current valuations need AGI to work. They just need enterprise and personal spending on AI to keep rising exponentially for a while.

We could absolutely wind up in a case where these companies are fairly close to a trillion dollars each, but not multiple trillions, at a reasonable multiple, all without AGI.

It being a bubble doesn't depend on whether it's useful or not.

It depends on whether the financial expectations on return on investment are realistic. I don't think they are. There's very strong fear of being 'left behind' and 'losing the race' so companies are ramming down the path of putting AI in everything whether it serves a purpose or not.

All these things we've seen in previous bubbles. In many cases the underlying tech was super useful and did change the world. Like the internet and the dotcom crash.

So is AI here to stay? Sure. Yes it will change the world. But no I don't think investors will be able to buy a private island next year. So much money is put into this that it's clear it won't be made back soon. Especially with the secondary effects creating (hopefully temporarily) unemployment etc.

Because businesses can spend a lot of money on AI instead of employees. But who are they going to sell their products to in end end? All those unemployed masses aren't going to be buying much.

> The entire premise on whether it is a bubble depends on if LLMs can be more than a next token prediction machine or not

A theoretically perfect next token prediction machine has limitless utility. Predict the weather, stock market, lottery number, the next step of groundbreaking cancer research, etc etc. That's not the argument against current LLMs being capable of being AGI.

This is not how LLMs work.

I think you misunderstood the point. The point is that "next token prediction is not enough" is not true; in fact, it's plenty enough.

The best next chess move is ___

The best stock to buy now is ___

If that is all you have then the next chess move will be random (or whatever token is more likely in isolation), or the stock prediction will be the same.

For the chess example to work you need the current board state in the context or the move history. How an LLM performs will depend on what it has learned across the transformer layers and in the residual stream, as well as how much chess data it has been trained on. -- NN models like Leela and Stockfish have been trained on thousands/millions of games and have a game-specific state space representation.

The stock prediction example is similar, but doesn't have a consistent state space in which to make that prediction. It will need information in its context (current news articles, stock history of all stocks, etc.) in order to be able to make a prediction. Its performance will depend on whether it has been able to learn factors that make it possible to make accurate predictions.

You are completely missing the point.

The point is that next token prediction is potentially very powerful. Nobody's claiming that LLMs can predict the market or win a chess match.

But the argument against next token prediction as a powerful tool is void.

It's the transformer architecture (token embeddings and cross-token encoding) and residual stream (cross-transformer encoding and token differentiation) that make LLMs powerful tools.

The key question is not whether this architecture is powerful. The key question is how well these models can represent things outside of their design.

With chess for example, are they actually learning the rules and strategies of the game (like AlphaZero/AlphaGo, Leela, etc.) or are they just relying on a large corpus of chess games? For example, a lot of early game moves will have been played out and the best moves will likely be the more frequent moves in the training set. So LLMs could do well early game based on that game corpus.

I think that you fundamentally understand "predict" differently from those making the argument.

> The stock prediction example is similar, but doesn't have a consistent state space in which to make that prediction. It will need information in its context (current news articles, stock history of all stocks, etc.) in order to be able to make a prediction. Its performance will depend on whether it has been able to learn factors that make it possible to make accurate predictions.

This also presumes that accurate predictions are possible.

A theoretically perfect next token prediction machine is just an LLM with a temperature of 0, which performs worse than one with a non-zero temperature (i.e. imperfection)

The "prediction" has nothing to do with predicting the future, it just means choosing the most likely next token for existing human text

No that would be a pretty terrible next token predictor. It would perfectly match the training data, but the training data is just a sample of what we are really trying to predict: human language, knowledge, and thinking, which is inherently probabilistic. Perfectly matching the training data is a pretty good sign that you've overfit.

A theoretically perfect next token prediction machine would be uncomputable.

> Adding reasoning agents, agent swarms, and so on to enhance the results from what is essentially just predicting the next token still means this technology at its core only predicts the next token. It is not AGI and unless something fundamental changes, it never will be.

A lot of people have dismissed LLMs four years ago for this reason. "It cannot possibly do coding since it doesn't think, it just predicts the next token." On the other hand, Scott Alexander did say in 2019 that next token prediction is a brilliant idea and might cover a lot, including math proofs [1]. So I wouldn't say it is impossible for either 1) LLMs to be all the way to AGI or 2) LLMs to become creative enough to tell us what the AGI architecture is.

[1] https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...

> Just look at what they have done to computer science already.

I do. And it’s not good.

Is there any evidence that you yourself are more than a 'next action predictor with a multi-modal context window'?

I don't think the nature of 'predicting the next token' is the problem. One could argue that any organism works the same way. The problem is that we've had million years of evolution to stabilize the platform such that the important parts of it work without that many hallucinations -- and even then that sometimes fails and you get failure modes like e.g. schizophrenia, where the context window gets confusing and the 'next actions' don't make that much sense anymore, seen from the outside.

An LLM never experienced evolutionary pressure the same way. There's no fundamental difference between how it was incentivized to answer correctly when asked about how many Rs there are in strawberry vs whether it is safe to leave a house through the second story window, even though for any physical organism, those two would be _very_ differently weighted during training.

We have a world model.

> Is there any evidence that you yourself are more than a 'next action predictor with a multi-modal context window'?

I love the age of LLMs because people will post stuff like “the fact that I absolutely, unqualifiedly, fundamentally do not understand two separate things indicates that I understand at least one and maybe even both of them more than you do“ and then do that cool move where they pretend they were gonna high five you but don’t

it is a golden age of Posting

It's because of the proliferation of "insight-posting" - people thinking their little insightful "aha" thoughts are unique or have any kind of value at all.

However, insight born of knowledge ignorance is worthless. I really don't care for your great insight about nuclear fusion if you're not a nuclear engineer or physicist working on nuclear fusion, because your insight is just a hypothesis in the void that is your ignorance.

Is your post any better? You attack GP's character without addressing any of his arguments. You could have provided evidence for why you think his speculation is wrong, or tried to point out how his logic falls apart.

> You attack GP's character

I said the move was cool ???

I'm not even sure which part of my post you object to, or why

I never understand the “only a next token predictor” argument. I do share the view that something fundamental about intelligence might be missing in LLMs, but I don’t see how this argument captures anything. If you can predict the exact next word Einstein is about to write, every time, you’re as intelligent as Einstein! It seems very clear that the LLM is also “thinking” about where the output is headed after this next token.

I do hear this position quite a lot though so I’d be interested to hear more about what I might be missing.

Next token predictor is not a very good argument. A submarine does not need to be a fish to swim.

Saying it is a 'next token predictor' should not diminish what LLMs are capable of. It is incredible what can be achieved by a simple rule like this.

Look what we built with only 2 digits: 0 and 1.

[delayed]

LLMs cannot, by their core design, predict next word correctly every time. That is the main thing you are missing. Another thing is that I know that I am not predicting next word based on othr words in my "trainin data", there is much deeper understanding of most subjects that I am using to "predict next word". This may not be true for everyone, I cannot know for sure, but it might be possible that there are people out there who are just biological LLMs, and in that case I am certain those people will not be able to predict next word Einstein would say either.

> If you can predict the exact next word Einstein is about to write, every time, you’re as intelligent as Einstein!

Einstein is not hold in such esteem because he was really could at spewing out text or formal proofs or theorems, but because he was able to invent something no-one else could. Some people say he advanced physics 100 years wuth his theories in relativity, whether it is true or not.

So even if you can predict what Albert is going to write next, you are still merely copying old Albert. They are amazing at processing large amounts of data, naturally, but creating something new and surprising we have yet to see evidence of (the solutions of the millennium problems seem to be brute-forcing a solution based on fuckton of data to me, but I am not a mathematician so I guess it is up in the air still).

The main criticism is, imo, that these models cannot think and have no logic. This is proven time and time again. Every day something new comes out, like counting Rs or the ”drive or walk“ test etc. These are the ”fixed“ or rather band-aided on a case by case basis, but nobody seems to be able to fox the underlying problem.

Having done my master‘s thesis on Transformer LMs, I think this is a fundamental limitation of the architecture.

These models can solve math problems and then fail to count the number of problems they solved. Not being able to count to 68 is not intelligence.

It seems you are just gatekeeping intelligence.

"Not being able to quickly reference the sum corpus of human knowledge across all fields, or contextually learn new languages in moments, or grasp the overarching architecture and modules of a million+ line codebase is not intelligence."

This would be a pretty ridiculous statement don't you think?

You are operating on a narrow, human-centric view of intelligence here for some reason... while I bet you would say an Orca is intelligent. So what drives you to place an LLM so low on the intelligence totem pole? This is clearly bias.

The fact that it is programmed is irrelevant. The fact that at it's core it is an ingestion of human data and an output of a prediction of more data doesn't declassify LLM's as intelligent. Unless you are willing to put that same lens against how humans learn. (you aren't)

Not that these distinctions are meaningful in any way, by the way, in light of raw functionalism. I am just annoyed by the clear bias.

Why would being able to quickly reference all of human knowledge be relevant at all? Intelligence is not about speed, it is about correctness, which is precisely why it's concerning that frontier models keep stumbling on things that should be piss easy to get correct.

Else by your logic the exact same model running on hardware that's half as powerful is somehow half as "intelligent"

And no, humans and other species don't learn the way that LLMs do. That's fairly obvious even if one isn't a neuroscience nerd.

It's relevant because it is another type of intelligence I was highlighting.

> Intelligence is not about speed, it is about correctness

What...there is a whole realm of correctness in the example I gave. Just because I didn't say it correctly references human knowledge, you presume correctness isn't implicit in that?

You don't know what you're talking about here. I was making philosophical and neuroscientific references that you are just blowing past for no reason. Lol

“It” is not “thinking” for christ’s sake, it’s a massive statistical regression curve fit to a massive, frozen library of text. Unless you’re thinking of “thinking mode” which still is just a template that tells the model to ask questions, which are asked one token at a time, turtles all the way down.

This may be news for some, but they actually train these things with RL now, not just pretraining on a library of text.

The thinking part is still not a “natural” or “organic” process happening during inference.

> I never understand the “only a next token predictor” argument

Yeah, that's typically the case for people who have no clue about LLMs or AI.

“Predicting the next token” is a bad argument, as it completely ignores all the emergent properties of transformer models. It’s like saying that discreet electrical impulses from sodium/ potassium transporters is the beginning and end of what neuroscience describes.

The entire premise on whether computing is a bubble depends on if computers can be more than a bit-flipping machine or not. Adding operating systems, compilers, distributed clusters, and so on to enhance the results from what is essentially just flipping bits still means this technology at its core only flips bits. It is not a general-purpose machine, and unless something fundamental changes, it never will be. There are certainly valuable applications for this technology, as we have seen in ballistic tables, census tabulation, and other sectors. But it is still not more than taking in voltages and toggling transistors between 0 and 1.

Software will never take off until we can show that it’s not just a bunch of simple instructions like load, store, add, and multiply.

But despite getting better and better at not making stupid mistakes, the models are still just next-token predictors. There’s plenty more room for them to improve without a totally different architecture.

I wonder: Are you (or any human) any more than a next token predictor? Can you prove that you are any more than the input of your genes/environment, with a deterministic output?