The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?

If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?

Yep youre right, I don't see an eval on this.

I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.

At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.

There doesn't need to be specific narrow training data on every conceivable task.

I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important)

It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.

And yet I would expect that if it truly was generally good at thinking as you put it that it would be better at chess if you gave it a set of general rules/tactics and have it think through and look forward to decide its moves, but it is quite bad.

How do we decide which version of thinking a model is good at?

A house sparrow (Passer domesticus) doesn't have a worse brain than a human. Nothing does house sparrow thinking better than a house sparrow brain. Not even a human brain.

what about a house cat?

they clearly outhink the sparrow at sparrowing in order to predict what the sparrow will and wont react to, and how theyll try to escape the cat

It's not clear to me that a house cat is better at out-thinking a sparrow than a sparrow is at out-thinking a house cat.

Avoiding predation is a pretty small part of the life of a house sparrow. There is no reason to believe a cat brain would do a good job with handling the demands of flight (avian brains process visual stimuli roughly twice as fast as mammalian brains, for one thing).

But even if we focus on avoiding predation: would a cat brain in a sparrow body (a silly premise, I'll admit!) do better at avoiding cat predation than a sparrow brain in a sparrow body? I doubt it!

One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.

I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.

But how would it know which weights to update? It has 3.5 and 3.8 weights memorized in its own weights and can somehow figure out which ones are responsible for different skills just in its own reasoning trace? (Since it didn’t use any external tools)?

That’s pretty crazy because, as I said in another comment, I just asked ChatGPT how many r’s are in blackberry and it said 3. I know it’s not Astra and the app doesn’t tell me which model it is, but I’m still very confused about how a model can edit neural net weights but not count letters in a reasoning trace.

Edit: Oh I misread, it didn’t use tool calls for the first part but it did have a custom harness for the part where it edited a models weights. Still impressive but not existentially confusing anymore.

You don't think an LLM can perform tasks not in its training data?

Not one which can easily be represented as an interpolation of other tasks it was trained on.

Well, it can't, by definition.

Almost nothing I prompt into existence already exists, otherwise I'd just use the existing thing.

What definition is that?

model

The definitions behind the math of LLMs.

This isn't true.

A neural network can do new things outside the training data. Random initialization sees to that.

Whether those things are useful is a different question, but it turns out that yes, some of those things are useful. It also turns out that in a large-enough LLM if you give it some kind of idea of what you see as useful then yes, it can pick out the newly useful things.

Random init only exists to make each neuron distinct from other neurons. The weight is still defined by the training data, without the training data, these random weight don't give us anything.

The second thing you said is basically what's called fine-tuning.

Yes you are right about random initilization of course. Long day here.

For the second: No, not fine tuning at all.

For one thing in-context learning is enough for learning novel tasks[1].

Secondly, it can do new things outside that data. LLMs build models of the world[2]. That means it can model novel behavior based on that internal representation.

[1] https://arxiv.org/abs/2005.14165

[2] https://arxiv.org/abs/2305.11169

Not sure what you're getting at, in any of the cases you said it needs to learn if it needs to do new things.

Maybe it can generalize outside of some learning task you gave it, but still it comes from the training data.

"Generalization" also comes with some luck, since these models most of the time output complete non-sense when it comes to tasks it "didn't learn" specifically. Just because you got lucky with the neuron connections doesn't mean it has what it takes to generalize anything.

I don't think it can whistle while chewing gum, no.

I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.

In my opinion that's very handwavey.

Definitely see where you might feel so. I provided an analogy that may help anchor - why ML fails extrapolation is left as an exercise to the determined reader! :)

The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!

the most obvious example was counting the Rs in strawberry.

figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together

theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely

you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it

>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?

I mean, this question indicates that your understanding of how these models work is, indeed, lacking.

Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.

I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.

> emergent behavior

This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.

Every learned ability of any neural network is an emergent property.

Let's analyze the premises

1/ The internal users at labs and external users are completely different.

2/ The lab people use the models to create recursive systems, systems to create prompts or synthetic data sets etc etc etc. Way different mentality than "fix the bug, make no mistakes"

Is it really so farfetched to think they've been having models fuck with the weights of models? I think it's entirely possible, probably even likely as a way of generating permutations of synthetic datasets that will then be included or excluded

> Is there really enough training data on direct weight editing to give a model this skill?

OpenAI has been gearing up for RSI. Jakub Pachocki has mentioned this as a priority, even as recently as September 6 in public. GPT-5.3-Codex helped with its own development.

They have been experimenting with it for post training (GPT-Red is a documented model-to-model training loop) but it's also likely there's pre training for it.