I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enourmous downstream effects that come with interpretability.

Good points - I’m still shook

Oh for sure, it's still really cool. A few months ago I had agents do some interpretability work on GPT-2 small, reproducing the circular weekday/month features from Engels et al., and it's wild how far you can get with just a bit of prompting, given I know very little about LLM internals myself.

It's like this in every field. If you don't know something, you can ask an AI and get blown away by how much it knows. I can ask it which tools I need to cut a dovetail joint and it'll impress me.

We used to get that feeling from Google, and from books.

The way you put it sounds far far more impressive than the way GP put it though.

In fact qwen knows much better math than their final fixed model. The nonthinking qwen 0.6B parameter model had 55.2% correct on math-500 vs their final improved 48.1% fixed-reasoning model, both with a 32k context window. The original qwen reasoning model had 77.6% accuracy on same test, though qwen recommends using a much larger context for these benchmarks. I wonder if their broken baseline run with 7.9% accuracy is a vllm or setup bug, but didnt dig into it any deeper.

Cool work anyways.

Qwen in reasoning mode is ridiculously anxious and ends up going in circles. It's not just math, it's how it behaves in general.

Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.

To be clear, the tuned model here performs worse than the same qwen model performed before RL, which was also not going in circles. It also performs much worse than the RL model that goes in circles. One could argue that the reduction of context may be worth something, but it appears that the final model is worse in all respects than the model at a much earlier stage of training.

You can ask Astra to study 2 or more models and ask it to import their layers to the Qwen model. With this will it not gain more than what it learned with RL?

I suspect the model was just hitting 32k context limit because it is trained for reasoning with 256k tokens available, not 32k

So it's like a model debugger?

Model psychiatrist is also fun, but probably less appropriate.

There’s something here…

It’s editing behaviour more efficiently than a psychiatrist, and is in ways acting as a brain surgeon but brain surgeons don’t focus on behaviour and thought loops…

Depends on the brain surgery... a lobotomy at the extreme end. Seizures could be thought of as a loop of firing neurons too.

It's a programmer that can debug, the model weights are part of the source code. The whole thing is just a program in a weird language and language models are good at translating across all sorts of formats. That's what they do when writing SVGs, generating 3D models or even directly outputting 3D models.

And they can write tools that help debug better than demonstrated in that blog post.

In my interpretability tinkering, Fable was the first to be able to write a program that can transpile from a lispy syntax directly to model weights of small neural nets, including transformers. Yesterday, I pointed out in a post that for many kinds of small AIs, you can avoid ML and have the LLM directly write the small neural net, compact decision tree, AD Tree or bayesian network. Since it's "hand"-written, it can be much more efficient and capable than training one of a similar size.

true but I think that is the first step now you distill a smaller specialized model on an improved Qwen model for specific tasks at least that has been my layman understanding of what is possible with these LLM

Yeah, I think the multiple experiments muddy the water.

Early experiments show one-shotting direct weight interp, which is crazy to me. But this is IIUC only on early layers.

The later math result gave the LLM the ability to run forward passes and inspect activations, and as others have noted is more of a classic steering vector intervention. Impressive how low to the metal this one was, but it’s a lot less so than the openers. Plus, as you say this is just bugfixing a known flaw of repetition, by finding the “repetition direction” and top triggering neurons, then rewiring the neurons to fire suppressing repetition instead. This is not a new technique and doesn’t shed any mechanistic understanding on how maths works.