Astra can read and write weights to improve Qwen

Re:Astra can read and write weights to improve Qwen by nutanc on Tuesday November 03, 2026 @06:53AM (#50350854)

I don't think weight space interpretability is the only thing. I think it's the fact that Astra(and apparently Opus 5.5) can directly read the weights and find patterns and make changes. Writing code for interpretability is one thing. I think this approach shows models now understand weights directly without tools.

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by yurimo on Tuesday November 03, 2026 @07:35AM (#50351247)

I think it is cool too, but with a caveat. For the large model (Qwen), their Astra used a Python environment, activations info, and iterative feedback to find an edit, it is not the same as looking at just pure weight numbers.

And in small MLPs they used, the I/O representations are kind of made to be more understandable because they explicitly encode objects and coordinates for games. If you set it up like that you can see some patterns yourself in the first/last layers without Astra. It is very different from interpreting distributed, entangled computations inside a large model. So I think the paper shows promising capabilities in weight-space analysis and editing, but not a general understanding of large models from their weights alone.

I'd love to see us reach that point, I am for one working in this area, but I don't think these experiments establish it.

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by christopear on Tuesday November 03, 2026 @07:40AM (#50351303)

Not quite true. In earlier examples (like Pong vs Pacman) the models are quite small and a lot of the topological structure can be read from the input weights. Whereas, in latter tasks, they give the model a harness:

> We give Astra a... Python session, like a notebook, with the model’s weights loaded. Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512)

It's interesting that they can now understand weight patterns, but not much more interesting than LLMs being able to read/write directly in Base64.

However, I think this likely indicates that OpenAI and Anthropic are now including model weights directly in their training - rather than an emergent property as the blog suggests. I think this is the more concerning finding - especially if doing so is in a bid to find methods for RSI.

[ Parent | on HN ]