yurimo

I think it is cool too, but with a caveat. For the large model (Qwen), their Astra used a Python environment, activations info, and iterative feedback to find an edit, it is not the same as looking at just pure weight numbers.

And in small MLPs they used, the I/O representations are kind of made to be more understandable because they explicitly encode objects and coordinates for games. If you set it up like that you can see some patterns yourself in the first/last layers without Astra. It is very different from interpreting distributed, entangled computations inside a large model. So I think the paper shows promising capabilities in weight-space analysis and editing, but not a general understanding of large models from their weights alone.

I'd love to see us reach that point, I am for one working in this area, but I don't think these experiments establish it.

I don't think I am. I have cited ROME as an early example of weight editing in large language models, the research has moved on since then. I think you are misinformed or don't know what has been done previously and extrapolating the results in this paper. In exp 1, you can recover a lot of the game geometry simply by visualizing the first-layer weights with numpy, which I actually did this morning, while reading the paper. In exp 2, the shapes are visible in the output-layer weights. The authors themselves acknowledge that these experiments exploit this. For 3, Astra uses Python scripts to run the analysis using head-composition techniques from the Transformer Circuits thread, once again authors acknowledge this.

In 5 and 6, the model designs game strategies and implements them as neural weights, we already know that models can do the design for simpler policies and Go heuristics are very familiar which the model first implements with python , program-to-weight compilation has substantial prior work, including Tracr (2023), a paper I particularly like: https://arxiv.org/abs/2301.05062. What matters here is automating the process rather than a new method. So "no human and no technology so far" makes no sense here, if you read the research. All of this is supported either by their experiment setup and citations, or prior work.

Aside from that, I think it is ironic that you cite a solution to the Navier-Stokes equation as some counterpoint to brute force, when that solution was found by scaling the agent swarm to massive proportions with tools and massive parallel exploration of that space, augmented by prior research on the subject.

To be clear, I think it is a cool work, but I think the part that matters is that at these smaller scales a current SOTA general purpose LLM can autonomously (and with help) apply these techniques or a form of them. That's worth investigating, I conjecture it is due to specialized post training on a custom RL env for mech interp.

It depends on your interest, since even though it is a subfield it is diverse. Relevant to this work here is a general overview: https://arxiv.org/abs/2603.10090

Also check out Tracr: https://deepmind.google/research/publications/22295/ Here they attempt to compile static programs into weights.

This is less of an overview but I remember reading this before, from the larger collection of posts: https://sidn.baulab.info/david_atkinson_project/

Maybe they weren't aware of the bug, steering is a good method to quickly improve the model, more popularly used to refuse unsafe queries. Qwen researchers do work on interpretability and try to make it practical in production: https://arxiv.org/abs/2605.11887 , I've read this paper this summer. Overall yes, when you edit weights what often happens is that performance deteriorates elsewhere, this generally scales with how destructive your method of editing is and how expansive your edits are. Analogous to if you use a jackhammer and touch every surface in the house versus using a small chisel on a door frame.

My hope that in the future new methods will be better at editing larger scales without significant deterioration, maybe even performance increase instead.

I don't know if it is due to recursion. I think it has more to do with what kind of mechinterp environments and data newer versions of models were trained on. It is actually, I'd say, a more interesting part about this paper as a whole. The authors tried SOTA models and found that they are better at classic interpretability. Some results, like identifying a game based on the weights, are less mythical if you understand that, i.e. an input or output dimension corresponds to a location or object channel, so structure in a matrix can have a readily recognizable interpretation. When we train a PPO agent on, say, a classic 2D game, the output is often a state of the agent in some board + environment reaction. For a human, this looks like an array of numbers; for an agent, it is a pattern corresponding to an output structure of Atari.

As for me personally, I am more interested in giving models better tools to understand themselves, so that they can derive a greater degree of learnable signal. More so, a bigger goal for me is for models to find these better tools themselves, hopefully better than human-made.

It does, one of my ongoing research papers is about it. But I would ask people to take a look at the established idea of Hypernetworks: https://arxiv.org/pdf/2306.06955v3 Modern version of the field comes from Google Brain around 2016 https://arxiv.org/pdf/1609.09106.

In general my personal research thesis is that current deep learning is much less malleable and neuroplastic as it should be, I got into interpretability because of this. Many focus on safety, but I prioritize it as a toolbox that allows for greater shaping of the underlying intelligence system. If we can fully understand how learning happens, where information is stored and how it is processed, unlike in biological brains, we can apply edits and transformations in digital ones. And if I can use a technique to edit weights based on some underlying reasoning about a system (with a caveat that it leads to something worthwhile), so can an agent. Hence self-improvement. There are many interesting implication to weight space research, and I very much encourage people to read about it. But it is also very easy to overstate what is currently possible, in fact I'd say better to know what is not so one can make it possible.

Scale is an interesting point here. Large overparametrized networks don't behave in the same way as mall toy models, this is one of the big caveats of many interp works. I expect naked weight edit will not work for large models. Why? Because an agents needs to know what it edits and reason on it. You could theoretically improve a model on some form of weight recognition, maybe it will work maybe it won't. But again, we come up against a cost/benefit dilemma, it sounds exotic and sci-fi but likely impractical. Parameters of model are connected and relate to each other. Instead, current SOTA auto-interp argues that what we need are better tools for models to know themselves, so rather than direct edit we need to give models better understanding of large parameter geometries, then they can work at scale. This is much harder to do than applying a steering vector.

Respectfully to the authors and to everyone here, I ask you to actually read the paper before losing your mind, or at the very least plug it into Claude. It is a cool project but not as nearly as insane as people say. I think the blog writing and subsequent posting overstate the results.

Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023: https://rome.baulab.info/

Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.

Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.

On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.

There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.

Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.

I'm not going to lie, one big reason I as a researcher have been a customer of OpenAI is that on their Pro plan Pro-tier model in chat and codex are decoupled in terms of billing. Anthropic instead billed you fully. This was very useful for long mathematical reasoning for me and I could attach outputs of my simulations in the context.

I also recall reading the following: https://newsletter.semianalysis.com/p/anthropic-subscription...

I will see once this transition actually happens but if another provider offers more for less and a cost of switching is inherently low as most of project lives on the disk and not in cloud, then why not move? For my workflow pro astra has been very helpful, likely will be less so if it becomes far too expensive.

RIP to the legend. Thank you for making my childhood and making me believe in the possibility of the better future.

I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.

This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).

I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.

I'll say while this is not very realistic now, I believe there isn't a 0% chance of a rogue AI traversing the net and placing itself somewhere(as much as this sounds like a scene from beloved cyberpunk franchises). As models become more efficient and capable, and hardware becoming better, hosting a model should be feasible on smaller devices (by compute and memory). It is possible it can be done in a distributed manner, where distribution is possible on different devices running the same OS or through some interface to collate different OS types.

I doubt the agent will be coming after your smart kitchen toaster, but a cluster with weak security is possible and tasty. This year there was a pretty loud demonstration of what a runaway reward hacking will do. I encourage people to read up on "stop button problem" i.e.: https://www.youtube.com/watch?v=3TYT1QfdfsM

While I think the article exaggerates to a degree, is it really so improbable that an agent seeking to avoid a shutdown and aiming to complete some goal and to get a reward will not attempt to escape if it concludes that it has failed, and humans will kill the process and hence prevent it from completing the task?

Note: I think siloed sharded inference is possible. But unnoticeable large-scale training probably not, at least with the current architectures.

A thing that I think about is that what you have said kinda compares itself to current math crisis. If we are only focused on the outcome and not the path it took to get there (i.e. understanding mechanisms of how the drug works) we might lose additional paths for discovery that would have bloomed from that process. Maybe an approach that did not yield a usable drug had a mechanism that would aid somewhere else. Yet the failure would be discarded as this does not grant the outcome for a model that attempts to optimize. Same is for math.

As aside I agree with you, but also worth noting that long horizon reasoning is difficult and AI, as good as it is, cannot predict something like 5-10 year drug interaction. Can it heal the patient now but cause a more aggressive cancer to come back and kill them in 5 years? You might say it is cruel to deny a person this treatment but if the doctor and manufacturer don't know the long term outcomes would it not be dishonest to tell the patient they are out of the danger? My point is, when we accelerate, we still should make sure we won't accelerate into a wall.

This is very cool, thank you. I understand the choice of no account, but would have loved if my progress could be linked between devices, maybe icloud sync?

Recent recording from NYU Courant institute.

This is cheap. Plenty of scientists, many of whom are my friends, are working very hard on finding new therapies for cancer, they were doing it before genomic models came along and still doing it now. The amount of times something in the media is lauded as "holy grail" that is never heard from again because it either only works in mice or turns out to be toxic or 100s of different reasons is massive. In my opinion this attitude of putting rose glasses on is detrimental to scientific progress. People outside of cancer research routinely underestimate how hard it is to find a working protocol. I think it is better to have sober attitude because it allows one to see the limitations and challenges that need to be tackled, blindly hoping AI can solve everything and deliver miracle cures is exactly the attitude that lets people sit on their asses and do nothing.

I think we need to be honest here. Author is basing it on one small experiment of picking up a block, relies on an whole IK controller pipeline to do the job, and does not compare it to full VLA or WAM models. They then proceeded to extrapolate the token throughput (mind you not the same as controller throughput) into supposed 2029 timeline, from one example.

Code as policy is a bad interface in my opinion, but VLM planning has promise. This has been tried in 2022 https://say-can.github.io/, and recently reformulated in https://lianegalanti.github.io/Pigey/

Thing is even recent Gemini Robotics 2 argues for architecture that has a VLM planner and then a VLA/WAM controller + a local small VLA model when connection disappears. And recent SOTA architectures rely on hierarchical design. I think this might be a sensible way to go about it. If you were to train GPT-X on robotics data and to output actions, congratulations! you've just made a VLA. It is enticing for people to just wish for one architecture to do it all, which is why we get stuff like this. I think there is a lot more to gain from modularity and we should not be afraid of specialization.

I think it is important to divine what currently AI is good for and what it is not even in such verifiable environments like math. Current hyped announcements about breaking conjectures are notable and are a marker of how much improvement was made. But as I read them, and maybe I am wrong, I see it as a large model+ harness executing a broad brute force search and trying solutions until something sticks. There are lots of problems like that and they should be solved, as often they are perhaps less important or overlooked, or just a slog, any field of research has these, math even more so.

However, this is very different from inventing new mathematical machinery that allows to break old problems, I think it will be a while until AI will be able to do it if at all. For now I think we will be moving to a symbiosis where an AI cracking a problem and giving a solution, inspires a human to invent new techniques.

Or use sticky notes! My friend annotated her favorite book with notes for me and it was awesome.

Personally found newest paperlike to be a lot like glass still, you barely get that paper feel at all and I hated it. They tried to keep clarity of the screen near glass level, but that affects the feeling a lot. Eventually I did some research and found that Japanese screen protectors that give you a feel of Kent paper are excellent for writing, so I wholeheartedly recommend those. You can find best rated ones by looking at top rated ones on jp websites, I have bellemond one.

I was considering getting M5 Max mbp with 128GB, but at that price it is just ridiculous, 64 version costs now roughly what 128 was before and that was a stretch for me. At this point might as well stick to my ol reliable M1 Pro mbp.

Edit: I'll say that now it seems also very hard to justify buying top of the line apple hardware for the enterprise. Getting top laptops for just a team of 10 people now means extra $20k just for RAM on top of already a higher base price.

Wow I'm old, I still remember working with YOLOv2.

I'm pretty sure this is not the use case at all but man do I miss bootcamp. Even for games if we could just run linux without a need for crossover, gaming on mac machines would be a dream.

I personally don't view coding agents making software as "software gotten better" you are comparing a tool and the end result, these are two different things. Agent you use going down and your product going down mean two different things to you customers. I will not deny that we made incredible progress in coding and hell, even design over the past 3.5 years, this technology is here to stay.

That being said while I agree that measuring better quality of software is vague (part of the reason it is hard for models as well), there are universal things I believe every engineer will agree on. Reliability, uptime, customer feedback, legibility of your engineering, performance, these are things we often optimized for. Google Maps is a bit of a strawman because neither of us (unless you work on it), knows how much agent code there is, I think it is likely that it's little since it was working fine prior to 2023. I could bring up github reliability as an example, given how much copilot usage they promote at MS, but once again only folks there know for certain. I do, however, see scores of various AI powered SAAS that looks like it is in a perpetual MVP state. I think you are right in that even if agents give us "good enough" results and we can swallow failure rates and our increasingly lesser understanding of what we, or more so model, created, then it is still progress overall, but this is progress not to human-AI collaboration but to AI-only engineering IMO, this is good or bad depending on how you view the future.

I'm a scientist and most of code I currently write is somewhere on the intersection of critical software and machine learning, squaring these two is not easy and I guess the way I was taught to reason about engineering informs my opinions on this. Maybe it's just a matter of time before codex can help here in an unconstrained manner as well, but I am skeptical at the moment.

You can't really do that here because one of the key arguments for this, as people in the thread focus on is "1/10th of time" estimate, the comparison with humans is here already, albeit it is just an estimate and no actual comparison has been done.

This is a problem of conflicting incentives that exists today in my opinion. Companies will market greater human-AI collaboration in science and engineering but focus on releasing things like this where it is clear that downstream goal is complete agent ownership over the product, from inception to testing to monitoring. Maybe the speculative future agents will use their own very efficient language to code that won't be readable for people at all. They focus on agent code being readable by agent in the article, as you've said. But in my mind in at least near future, there is a case where your prod will break, you won't be able to understand it or the attempted fixes. Maybe agent will fail to fix it at all and start a massive rewrite. In any case is this different from kicking technical debt down the road along with worse interpretability of what you have built?

I do think there is a way where agent can write great solid code that we can read, but with the way LLMs are built this requires something new in terms of reward that accounts for "taste" and constant refinement so it might take more than 1/10th of a time to produce something good.

I think the telling part is in this line:

> Because the repository is entirely agent-generated, it’s optimized first for Codex’s legibility

I asked a question from a perspective of a human engineer, as in, I will have to read the code and understand, fix it once it breaks. OpenAI approach is opposite, even if it is breaking it is the agent that will be doing the fixing, millions of lines and inelegant designs don't matter because human readability doesn't matter. In any case you use more tokens so you fork over more money.

I will say, however, that IMHO there is objectively bad and good code in terms what it can do and performance, if I can do the same thing in 50 lines as opposed to 1000 lines, this difference still matters for the model. Smaller context usage, better approach that informs downstream generation.

What I still can't understand is why is massive amount of code generated is a flex? I don't feel that software has gotten a lot better in past 3 years, only sloppier. It's surprising to me that people who know about reward hacking choose a simple objective like lines of code generated as a signal for quality. I'd argue you have to optimize for less lines generated as possible while secondary optimization should be readability for humans. I suspect it's not seen as a problem by providers because more lines generated means more tokens used and hence more billing put out on customers.

And if I am working on an existing codebase then isn't a good commit often a negative sum between added and removed lines? I don't want to bloat my codebase but make it more polished and elegant. After reading that I wonder if what they have done could have been accomplished for a far fewer LoC budget.

Great stuff, thanks for posting.