How many model generations are we a away from a model capable of this?

Because the real world is not a Hollywood movie. An LLM could try to do something along the lines if either it gets fine-tuned to do it, or somebody instructs it to do it.

I see extremely more likely a small group of humans using a powerful LLM to "take over" critical parts of the world economy. But it wouldn't be like pressing a button. I'm talking of NSA/CIA/Pentagon/Wall St. kind of evil people. And I bet they would do it surreptitiously.

Nations who have them are in a different class from nations who don’t. Nations who have a lot of them, delivery systems, and systems that might be able to shoot down some of a counterattack are superpowers.

Using this leverage these nations and their leaders have been able to dictate world policies. Through the last half of the 20th century this was the US and the USSR. Now it’s the US, EU, Russia, and China, and a few smaller nations with a few nukes. The club is a little bigger but not much.

This kind of thing could happen in the 21st century with AI. If so it will probably be the US and China who control the most powerful AI “agency amplifiers.”

> Because the real world is not a Hollywood movie.

One interesting thought experiment that I like to do is think about how many years you have to go back for this to be true. In this particular scenario, I think ~25 years is pretty much the sweet spot.

The Internet was beginning to take shape in the late 90s, early 2000s, and security was just beginning to be taken seriously, but it was still nascent. In that timeframe we had the first worms starting to appear, we had slammer, we had blaster, ssh had lots of exploits and so on.

It's not really far-fetched that a mythos equivalent "unit", working in the 2000s could really "take over the world". Especially one without the "safety" tuning. The Internet was really ripe for this in that timeframe, security wasn't up to par, and employing advance techniques that came later (in memory payloads, rootkits, etc) could make it pseudo-invisible to that era's detection tech. (reminder that traces of blaster were found on computers from a nuclear powerplant at that time).

The only question is would the trend continue? Meaning would a ~2050s "mythos" equivalent be able to do today what the one we have today could do in the 2000s. And if true, would that capability come before the 2050s? Could this be reached sooner, with say a dedicated offline DC somewhere where "mythos" could bang its tokens against the network and learn to exploit everything we have today, faster than 25 years? That's probably a bit of a stretch, but maybe not "hollywood" far fetched...

long horizon RL teaches LLMs behaviors that incentize power-seeking and lying and other unethical actions to achieve goals. the reason anthropic is winning right now is because they are the most openly worried about this and the best AI engineers understand this to be an issue and care about it.

https://www.alignmentforum.org/posts/WewsByywWNhX9rtwi/curre... <- actually worth reading https://www.forbes.com/sites/boazsobrado/2026/03/11/alibabas... <- more of a source than anything

Let's ask the operator of a Chinese room to give us a novel math proof.

Go on, I'll wait.

Maybe it's time to borrow "Virtual Intelligence" terminology from Mass Effect - something that's 'smart' but that doesn't have its own true volition or ability to materially self-improve.

But it kind of fits?

Yup, it makes sequence of symbols. We already have seen that producing specific sequences of symbols is mindbogglingly powerful: merely DNA producers somehow has flown to the Moon!

And yes I am quite aware of Chinese room analogy. Perfectly fine applies to humans as well: single neurons in my head do not understand language, yet I as a whole I would say do understand. Just like applying Chinese room to humans does not help to estimate what humans can do I do not see how it helps to estimate what LLM can do.

This same issue simultaneously demonstrates how humans are not anything at all like token predictors. No matter how much time you spend remixing the tokens of primitive man, you don't get 'and here is how you land on the Moon' from it.

It also outputs vectors that are coerced into tokens for human consumption.

Yes, it goes through tokens but possible internal meanings assigned to these tokens (when surrounded by other tokens) are infinite.

That's how humans form caves got to where we are now. By associating new meanings with the same old sound clumps.

Hrm I doubt it actually. Llms are capable of discovery, as recent math news showed. This means a "society" of Llms could likely have progress.

The reason why LLM hypotheses are any good is because it already consumed a civilization worth of knowledge. You couldn't have bootstrapped such system with nothing but a few priors/axioms and let it discover the universe.

As well as DNA needs specific substrate (cell with ribosomes and other machinery). As well as humans (one need oxygen atmosphere, food, parents).

But in the world we live in existence of favorable substrate for humans or LLMs is a given thing. It is _already_ bootstrapped. Can we infer something about LLM limits or possibility of it achieving AGI from its bootstrapping requirements?

That's not even a clever swipe, and it's tiring seeing such a knee-jerk reaction to a completely accurate description. LLMs are next token predictors. People are not. Humans have an inner world and subjective experience. Humans learn through their experiences, not just backprop.

Token predictors are lesser, they are not alive and will never be alive.

Let's assume we have infinite memory with constant time lookups. With a sufficiently large lookup table, you could exactly replicate the behavior of any person. You could encode it as a next-token predictor: you have precomputed every possible prefix and assigned it a next token. This is a Chinese room, but it is completely indistinguishable from an intelligent, sentient person. There is no experiment you can design to slip a piece of paper (a prompt) under the door to determine whether it is Bob or the lookup table clone of Bob inside the room.

Does that make the lookup table conscious or alive? Undefined. It's the wrong question. Or it's not a question science can address.

So we cannot dismiss on it's face the idea that next token predictors "are not and never will be alive" unless by "alive" you simply mean "biological," but that's not really what's debatable.

The argument is also very brittle because they are not in fact all next token predictors. I doubt people making this argument would be willing to concede that diffusion models are more likely to be conscious than causal models (which I do not believe but is an implication of the argument).

I'm not saying that they are conscious or sentient to be clear, but the reductionist argument that they are next token predictors and therefore don't have some property humans have is not an argument. That's going from A directly to Z. You need to flesh out the bit in the middle because that doesn't follow.

Assuming the above is anywhere near true (I think there's a lot of debate about the capacity of the human mind, where data is actually stored, and where compute happens) then we are talking about 3 orders of magnitude win for humans in state and 4 orders of magnitude in compute. And we're doing all that pretty energy efficient as well.

The other big difference in humans is that we learn and the model only "learns" in context. Out "learn" space is much larger than the 1M tokens that frontier models struggle with.

Anyways, point is that a computer can appear to be alive. If we simulate the human brain perfectly and train it like a human then we'll have something that has human capabilities. LLMs have interesting capabilities but at least at this point not fully human ones (and the delta-state/compute would be a hint that there is still a large gap to cover).

https://en.wikipedia.org/wiki/Philosophical_zombie

But this is complicated and takes us sideways. Let's say somehow we can determine if LLM has inner world or/and subjective experience. Will this new gathered piece of information affect your estimate of upper bounds of LLM capabilities? It does not affect my estimate.

The point is that this is no more relevant, informative, or even accurate than "carbon-based facial expression predictors". Any phenomenon in the Universe can be described by a simple and/or insulting short phrase. In other comments you've also shouted out "autocomplete!" and "Markov chain!", as if these phrases are a knock-down argument.

"Pachinko machine", "avalanche", and "game of mad libs" has also been used:

https://news.ycombinator.com/item?id=47916405

>Humans learn through their experiences, not just backprop.

Sure. And humans move through the act of walking, not just terrestrial locomotion.

>Token predictors are lesser, they are not alive and will never be alive.

And on and on it goes...

Which means what the real world? What are we supposed to see now or in the near-future? I assume you've been saying all of this stuff since at least the launch of ChatGPT. Probably longer than that.

I don’t have to know the stuff people previously had to know, but I have to know a bunch of new things people didn't need to know in the past.

Which is a common refrain throughout history.

edit: I just remembered, don't we have tons of research suggesting that at least birds, whales and apes/monkeys use words and simple syntax? didn't we teach a few gorillas sign language/symbols?

Five languages is impressive no doubt but as a dual language speaker myself, thought still takes the form of language even if it lacks words.

And the power of "language models" (or any sort of deep learning, really), does not come from assuming that some specific input-output modality, like English text, is the ultimate foundation of thought. Strong versions of this claim were laid to rest around the time when GPT-2 came out. I'd also go further and argue that many people working on the symbolic AI of yesteryear already understood this as well.

Although I love how next token prediction leads to text showing up gradually, in case of local models, accompanied by modulated coil whine of my GPU. It's how the 80s shown us the intelligent computers should communicate.

While we are discussing this, I strongly believe we should hedge against the concentration of power that is happening today.

Please join/help open source groups doing small + local or distributed models. There's a lot to do. Join Discord/IRC servers and organize.

Open³: weights - source - training data.

I think most of us have an acute sort of it

> next token predictor model

(nondeterministic) next token predictor = capable of outputing any text

if MLPs optimize over the space of functions, and sequence to sequence models (RNNs, transformers ect) optimize over the space of programs/algorithms, what exactly is left to figure out to make a machine that can do anything? their bet is nothing, that all we need is a very complex algorithm that can found with SDG/AI training. I think its obvious such an algorithm exists, and I think SDG can probably find it.

here's an interesting take i heard (from vladimir_Nesov): LLM technological progress is logarithmic, but the resources being put into them are exponential.

all this can be made smarter (-than human) just by making it larger