It's easy to trace that a bit sets a character in a file with absolute certainty; weights are in a highly dimensional space however, so even if ⍺v is good for one test it could easily regress for another.

I suppose the argument is that "editing weights" seems to imply a complex awareness of what each weight means, which is material for AGI hype. Really it's just doing very broad statistics, which is of course interesting but hardly unexpected.

I look at it at a slightly different angle. AGI implies broad capabilities including self-improvement without using another AGI. Being able to perform a “surgical procedure” on weights is a clear step in that direction. Direct weight interpretation may work if small model weights are included as training tokens but i doubt it’s useful for self-editing. On the other hand, using simple tools to interpret and edit it’s own weights might work.

Yes, the arguments in this thread seem to be around interpreting the word "editing". Is something consciously "editing" by tweaking the correct neurons/weight. Or is it just some linear algebra.

Though, even on re-read, it sure seems like Astra was making some decisions, otherwise what is this saying, that Astra knew how to download some linear algebra tools?

The weights ended up edited. It does feel a bit like the long of reasoning that claims LLMs don’t really think, while they’re doing novel mathematical research.

I think the distinction is relevant even if the exact line is blurred. Two extremes:

It would be extremely surprising for a model to be able to ingest all model weights in a prompt and output better model weights, with no tool calls. While certainly possible for small simple models (eg a 4-parameter perception), at scale this becomes difficult in terms of near-perfect recall at huge context size and requires a superhuman intuitive understanding of linear algebra.

On the other hand, it is extremely unsurprising that a model can edit weights when having access to unrestricted tools, time, and resources (including non-public resources that the AI labs have). The methodology is fairly simple: just train the model more on whatever better non-public datasets. Hell, just benchmaxx the smaller model, that would be "a method to edit the weights" that achieves the necessary results.

I think everyone would agree that these two scenarios are very different. Where the actual Astra weight-space programming result falls on this spectrum is debatable.

Astra was directly editing the weight, ie modifying the values without backprop.

It’s a bit conflated because the paper covered a bunch of experiments.

The early experiments did read/write the weights directly, no forward pass.

The math eval improvement experiment that got a lot of attention 1) fixed a specific known problem (looping) without actually understanding how the model solves problems, and 2) was allowed to do forward passes to check the results of the edits.

As others have noted, steering vectors are interp 101 and it’s pretty rare that there is an “act dumb vector” just waiting to be steered out of the residual stream.

If you directly[1] edited the bytes of a program, that is substantially different to if you got the source code from somewhere, edited that and recompiled it.

[1] Using a hex editor - not by waving a magnet at your harddrive or whatever.

That is better example.

Using the program analogy. Astra is looking at outputs of a program, and re-compiling it.

And this thread is kind of arguing over whether using Notepad is really programming, and others saying, Astra really used VSCode with some fancy plug-ins, so that isn't really programming.

No.

The neutral network was originally produced by training it in the usual way. So the training data is the "source code" and back propagation / gradient descent optimiser is the "compiler". That's what the human used to produce the network.

The weights are analogous to the compiled machine code. The LLM bypassed the compiler and edited those directly.