Astra didn't directly edit the weights, it came up with a method to edit the weigths?
So, it used another tool to edit the weights?
This kind of seems like saying, I don't directly edit the bits on my hard drive, thus i'm not really the one saving a file?
I suppose the argument is that "editing weights" seems to imply a complex awareness of what each weight means, which is material for AGI hype. Really it's just doing very broad statistics, which is of course interesting but hardly unexpected.
Though, even on re-read, it sure seems like Astra was making some decisions, otherwise what is this saying, that Astra knew how to download some linear algebra tools?
It would be extremely surprising for a model to be able to ingest all model weights in a prompt and output better model weights, with no tool calls. While certainly possible for small simple models (eg a 4-parameter perception), at scale this becomes difficult in terms of near-perfect recall at huge context size and requires a superhuman intuitive understanding of linear algebra.
On the other hand, it is extremely unsurprising that a model can edit weights when having access to unrestricted tools, time, and resources (including non-public resources that the AI labs have). The methodology is fairly simple: just train the model more on whatever better non-public datasets. Hell, just benchmaxx the smaller model, that would be "a method to edit the weights" that achieves the necessary results.
I think everyone would agree that these two scenarios are very different. Where the actual Astra weight-space programming result falls on this spectrum is debatable.
It’s a bit conflated because the paper covered a bunch of experiments.
The early experiments did read/write the weights directly, no forward pass.
The math eval improvement experiment that got a lot of attention 1) fixed a specific known problem (looping) without actually understanding how the model solves problems, and 2) was allowed to do forward passes to check the results of the edits.
As others have noted, steering vectors are interp 101 and it’s pretty rare that there is an “act dumb vector” just waiting to be steered out of the residual stream.
[1] Using a hex editor - not by waving a magnet at your harddrive or whatever.
Using the program analogy. Astra is looking at outputs of a program, and re-compiling it.
And this thread is kind of arguing over whether using Notepad is really programming, and others saying, Astra really used VSCode with some fancy plug-ins, so that isn't really programming.
The neutral network was originally produced by training it in the usual way. So the training data is the "source code" and back propagation / gradient descent optimiser is the "compiler". That's what the human used to produce the network.
The weights are analogous to the compiled machine code. The LLM bypassed the compiler and edited those directly.