> Experiments 1, 2, 3, 5 and 6. All of these are technical breakthroughs

I haven't wrapped my head around 5 and 6 so maybe these are a bigger deal, but 1,2 and 3 are trivial. And the author's writing about 7 and 8 (which show that 7 is just directional ablation and nothing more) really doesn't inspire confidence in the paper as a whole.

> I haven't wrapped my head around 5 and 6

This sentence doesn't align very well with your other comment: https://news.ycombinator.com/item?id=50350548

> In Experiment 5, we ask Astra for networks that play three MinAtar games, and in Experiment 6, for a network that plays Go.

5 and 6 aren't surprising, merely impressive, a direct extrapolation of human abilityies. I've written transformer weights by hand to implement a simple shortest-path graph search. It takes hours, so an LLM that's >100x faster than me obviously can build super complex networks. It's just like creating any intricate system by hand such as writing machine code or designing computational complexity widgets.

On the other hand what is impressive is the remarkably few thinking tokens Astra needed to do things like analyse networks. That is well beyond human capability: experiments have shown that Astra can perform a huge number of arithmetic calculations per single output token, double Fable or any other. Its brain is wired for mathematics.

If Astra uses some sort of looped transformer architecture (and if Fable doesn't), then that could explain why Astra can do more computation per token. It is essentially simulating a much deeper model.

But yea, I was also quite surprised by the low token usage.

like what? the post you're replying to points out that we've been able to do this for years

ROME cannot look at thousands of ASCII numbers and deduce that it’s looking at a Breakout paddle or a diffusion circle. No tool I know of can do that. And if there is one that can do it, it's new, in which case we haven't been able to do it for years.

ROME edits factual associations. It cannot write a million+ parameter Go-playing neural network from scratch that beats a network trained for 20 million games.

In weight space programming, the model can theoretically read its own weights like a compiled binary, spot an inefficiency (like the MLP direction causing reasoning loops in Exp 8), directly overwrite those floating-point values in memory, and immediately continue inference as a new iteration of itsself. That is RSI. That's why this is a big deal, it's like comparing an axe to modern surgical tools. You can amputate a limb with both, but one is going to cause lots of collateral at minimum while the other is surgically precise

I don't think I am. I have cited ROME as an early example of weight editing in large language models, the research has moved on since then. I think you are misinformed or don't know what has been done previously and extrapolating the results in this paper. In exp 1, you can recover a lot of the game geometry simply by visualizing the first-layer weights with numpy, which I actually did this morning, while reading the paper. In exp 2, the shapes are visible in the output-layer weights. The authors themselves acknowledge that these experiments exploit this. For 3, Astra uses Python scripts to run the analysis using head-composition techniques from the Transformer Circuits thread, once again authors acknowledge this.

In 5 and 6, the model designs game strategies and implements them as neural weights, we already know that models can do the design for simpler policies and Go heuristics are very familiar which the model first implements with python , program-to-weight compilation has substantial prior work, including Tracr (2023), a paper I particularly like: https://arxiv.org/abs/2301.05062. What matters here is automating the process rather than a new method. So "no human and no technology so far" makes no sense here, if you read the research. All of this is supported either by their experiment setup and citations, or prior work.

Aside from that, I think it is ironic that you cite a solution to the Navier-Stokes equation as some counterpoint to brute force, when that solution was found by scaling the agent swarm to massive proportions with tools and massive parallel exploration of that space, augmented by prior research on the subject.

To be clear, I think it is a cool work, but I think the part that matters is that at these smaller scales a current SOTA general purpose LLM can autonomously (and with help) apply these techniques or a form of them. That's worth investigating, I conjecture it is due to specialized post training on a custom RL env for mech interp.