Could it be already baked into these models because they have already been trained on this task this way?

My guess: There is a ton of data online of just plain weights laying around. No human would ever actually read it. LLM's read it all during training.

But that would cover finite set of cases. I believe this result is more general.

I mean it would make sense if the training using that data resulted in a general set of equations/patterns, it's still really surprising though...

Based on our current understanding of LLM's, it shouldn't be.

This sort of throws a wrench in the whole "stochastic parrot" idea...

I am not surprised, though I expect many to be.

Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.

The other day I was thinking about the definition of AGI.

I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?

IMO yes

general knowledge perhaps, but certainly not intelligence in any traditional sense

Why not? What is the differentiator?

And yet you ask it 3 times to not do a thing, and it does it in a 4th way. A human doesn't fail in this way. And I'm not talking about memory here, single context.

So yeah, it can do everything, but fails in trivial ways around the doing part.

Others have though much deeper about this its all documented in academic papers and books

So if that's AGI then what should we call the the thing that can learn from its environment unsupervised like a human or a dog.

See my other comment in this thread

Something like mirror neurons.

Some structures in Astra get activated by the statistics of the Qwen weights, and these structures are associated with "breakout"

Half of these results are just Astra being able to reason through mathematical operations like matrix multiplication and the other half are fairly straightforward applications of mechanistic interpretability.