In general my personal research thesis is that current deep learning is much less malleable and neuroplastic as it should be, I got into interpretability because of this. Many focus on safety, but I prioritize it as a toolbox that allows for greater shaping of the underlying intelligence system. If we can fully understand how learning happens, where information is stored and how it is processed, unlike in biological brains, we can apply edits and transformations in digital ones. And if I can use a technique to edit weights based on some underlying reasoning about a system (with a caveat that it leads to something worthwhile), so can an agent. Hence self-improvement. There are many interesting implication to weight space research, and I very much encourage people to read about it. But it is also very easy to overstate what is currently possible, in fact I'd say better to know what is not so one can make it possible.
Why do you think this is? I am very new to this research, but to me, the biggest challenge appears to be how information seems to be stored in superposition (making it difficult to change one thing without affecting others). But biological brains appear to exhibit the same kind of superposition storage as well, and yet they are very plastic.
My guess would be that the structure to allow for plasticity to work is a strong limit to how well you can tune weights to fit training data. My (doubtless massively oversimplified and way behind the state-of-the art) understanding of neuro-plasticity is something like "often used pathways get stronger connections". Which, even with significant tweaks, is quite different from the gradient descent used to tune neural networks.
It argues that our understanding of neurons storing information in superposition was just a limitation of our tools and us thinking about it wrong. Instead most neurons initially specialize, and then branch out into related concepts. In addition, understanding one part of the network helps better understand other parts. And with agents we can scale this to understand non-trivial networks.
While, if true, this would more directly help with interpretability, a better understanding could also help with improving NNs in general.
I'm reminded of how humanity recently finished mapping out the fruit fly connectome; this does not put us meaningfully closer to, say, doing surgery on humans to cure schizophrenia.
Russian native speaker, with approximately C2-level English proficiency. I am now about 85% confident that English is not his native language, and 35% sure that Russian is his first language.
Would that still be true for MoE models where only a small part is activated on a specific topic?
I suspect a lot of upvotes also come from people afraid of closed western labs running away with AI. Similar to other articles with positive news about open weight models.
If so, it is something different from what you've been working on, since the models that support it are limited to the very newest releases from the big-two labs. Thoughts?
As for me personally, I am more interested in giving models better tools to understand themselves, so that they can derive a greater degree of learnable signal. More so, a bigger goal for me is for models to find these better tools themselves, hopefully better than human-made.
From my experience with models when it comes to discernment about particular things, the models, even the best ones, still are not very good. The thing that we are most missing I think is acceleration in creating editing tools for humans that make the otherwise inscrutable both visually legible and interactive.
It was included in the prompt itself. The authors say they used claude code for the experiment, and claude wrote the prompt for astra, which included details of the looping bug. Then astra seems to have done h(looping) - h(loop fixed) and then gotten the usual steering vector.
_________
* The first author listed is a first-year Phd according to their X profile: https://x.com/rikiyatakehi/status/2117313742891327994.
But that's the content economy I guess.
Just because something is human doesn't mean that it is good
It forces people to actually learn and understand the tools they're wielding, so that they can withstand said rain with no issue.
The people getting washed away by it would be a threat to sanity at large otherwise.
That is not cool at all.
A phrase has been rolling in my head since all this LLM stuff started, you don't hate AI you hate capitalism. Every objection I've heard for LLMs are basically an issue of the benefits and power being concentrated in a small set of hands.
I had Claude train three language classifiers for short words in four languages. The network was just over 10k weights with a single hidden layer.
I uploaded 6 language word lists and made it train networks to fit random subsets of 4 languages. The languages were picked randomly. Claude could always identify which 4 languages were chosen and in which order. It told me that the language heads were easy to link to letter frequency and each language had distinctive makers. So claim upheld, an LLM can interpret small networks.
I then asked Claude to improve the classification accuracy by proposing an edit to the weights in the above networks. It's proposed improvement reduced classification accuracy significantly. The same thing happened if I gave it a deliberately undertrained network with clear room for improvement.
Also possible that it's leakage (which honestly does seem like the most likely explanation).
As a reminder, if you see incredible performance from any statistical model, it's most likely leakage.
My hope that in the future new methods will be better at editing larger scales without significant deterioration, maybe even performance increase instead.
(can be as maths heavy as possible/necessary to understand the topic)
Also check out Tracr: https://deepmind.google/research/publications/22295/ Here they attempt to compile static programs into weights.
This is less of an overview but I remember reading this before, from the larger collection of posts: https://sidn.baulab.info/david_atkinson_project/
A model has internalized and most likely generalized mechinterp research. Yes, it's expected and nothing insane. Just like internalizing and generalizing math research was expected and not insane. The results of the latter are... interesting though.
The fact you can give an LLM a raw dump of neural weights and it will hand-wire a bunch of changes on the fly is quite something. It's not so much about the OMG! RSI! for me personally, it's more about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien and we need time to adjust to that.
It would be interesting if asking astra to edit your model becomes a new way, like distillation, to train your model on the cheap by extracting OAI's model work.
But it's not what's happening here at all! Read the thing in detail and you'll see that it's not doing anything alien, it does exactly what you (or at least someone with at least some knowledge abouy neural nets) would do if you were asked to perform the task. It doesn't reason about weights in a magical way, instead it reads the easy to interpret parts or use directional ablation to tweak a model's behavior, exactly like how the ERP folks have been decensoring models to produce adult content over the past two years.
It's an interesting demonstration of ML knowledge and agentic ability, but it's not alien NY any means, and it's not actually surprising given how strong today's models are with agentic work.
When I say "alien" I don't mean "magic" and I don't mean "emergent new woo-woo scary ability". I mean, not human in speed/volume. No human can process that amount of raw bulk in that amount of time successfully. And indeed, it's not magic at all. It is enormously mundane actually. It reminds me of old computers being able to do arithmetic at insane speeds compared to humans. It's not magic at all, but to be honest, I have the distinct feeling of goalposts flying all around me. Being able to handle tasks of this complexity.. I don't know about you but I still have to adjust to computers being able to do this and that it's actually "not surprising". I guess I'm getting old.
I can see how you want to calibrate the enthusiasm I see around here as it can be a bit.. well, but it's not necessary for me. I'm not overly enthused by this because I don't care about the NN adjusting itself and RSI etc. I'm sure it's all quite basic but the fact it can do it at all is reason to be.. watchful, IMO. That is to say, I can appreciate the level of capability necessary to do this work successfully, but I do take your point seriously. It made me think, thanks.
The model being able to simply type out NN weights, on demand, and have it outperform one trained on 100M inputs pretty much fits that description, doesn’t it?
And being able to prove that knowledge is localized using tools, is also quite different from an LLM simply looking at raw weights and instantly pointing out that 18373 controls left movement.
It’s not a fundamental change in what we know, but a new emergent ability that surely warrants some excitement.
Experiments 1, 2, 3, 5 and 6. All of these are technical breakthroughs
Weight space programming is a holy grail of RSI research, and what you are talking about is very different from that, while the paper's proposals are exactly that.
It's similar in spirit to the Navier Stokes solution in that this level of LLM can intuitively navigate extremely large search spaces, rather than inefficiently bruteforce something
I haven't wrapped my head around 5 and 6 so maybe these are a bigger deal, but 1,2 and 3 are trivial. And the author's writing about 7 and 8 (which show that 7 is just directional ablation and nothing more) really doesn't inspire confidence in the paper as a whole.
This sentence doesn't align very well with your other comment: https://news.ycombinator.com/item?id=50350548
5 and 6 aren't surprising, merely impressive, a direct extrapolation of human abilityies. I've written transformer weights by hand to implement a simple shortest-path graph search. It takes hours, so an LLM that's >100x faster than me obviously can build super complex networks. It's just like creating any intricate system by hand such as writing machine code or designing computational complexity widgets.
On the other hand what is impressive is the remarkably few thinking tokens Astra needed to do things like analyse networks. That is well beyond human capability: experiments have shown that Astra can perform a huge number of arithmetic calculations per single output token, double Fable or any other. Its brain is wired for mathematics.
But yea, I was also quite surprised by the low token usage.
ROME edits factual associations. It cannot write a million+ parameter Go-playing neural network from scratch that beats a network trained for 20 million games.
In weight space programming, the model can theoretically read its own weights like a compiled binary, spot an inefficiency (like the MLP direction causing reasoning loops in Exp 8), directly overwrite those floating-point values in memory, and immediately continue inference as a new iteration of itsself. That is RSI. That's why this is a big deal, it's like comparing an axe to modern surgical tools. You can amputate a limb with both, but one is going to cause lots of collateral at minimum while the other is surgically precise
In 5 and 6, the model designs game strategies and implements them as neural weights, we already know that models can do the design for simpler policies and Go heuristics are very familiar which the model first implements with python , program-to-weight compilation has substantial prior work, including Tracr (2023), a paper I particularly like: https://arxiv.org/abs/2301.05062. What matters here is automating the process rather than a new method. So "no human and no technology so far" makes no sense here, if you read the research. All of this is supported either by their experiment setup and citations, or prior work.
Aside from that, I think it is ironic that you cite a solution to the Navier-Stokes equation as some counterpoint to brute force, when that solution was found by scaling the agent swarm to massive proportions with tools and massive parallel exploration of that space, augmented by prior research on the subject.
To be clear, I think it is a cool work, but I think the part that matters is that at these smaller scales a current SOTA general purpose LLM can autonomously (and with help) apply these techniques or a form of them. That's worth investigating, I conjecture it is due to specialized post training on a custom RL env for mech interp.
https://x.com/SchmidhuberAI/status/2117446822855888956
And btw it's not RSI:
a) There is no evidence that a model can weight-train a larger, more powerful model.
b) There is direct evidenct that a model cannot weight-train new capabilities to another model (see section "Additional experiment" also cited by OP, where they fail to improve Gemma 3 270M which doesn't loop).
No RSI without those things.
I do agree that RSI with this method requires those two things. But I view this paper as the opening salvo rather than defining the limits of what can be done.
And in small MLPs they used, the I/O representations are kind of made to be more understandable because they explicitly encode objects and coordinates for games. If you set it up like that you can see some patterns yourself in the first/last layers without Astra. It is very different from interpreting distributed, entangled computations inside a large model. So I think the paper shows promising capabilities in weight-space analysis and editing, but not a general understanding of large models from their weights alone.
I'd love to see us reach that point, I am for one working in this area, but I don't think these experiments establish it.
> We give Astra a... Python session, like a notebook, with the model’s weights loaded. Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512)
It's interesting that they can now understand weight patterns, but not much more interesting than LLMs being able to read/write directly in Base64.
However, I think this likely indicates that OpenAI and Anthropic are now including model weights directly in their training - rather than an emergent property as the blog suggests. I think this is the more concerning finding - especially if doing so is in a bid to find methods for RSI.
Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023: https://rome.baulab.info/
Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.
Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.
On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.