that is why this is crazy,
give a human a legacy codebase and 5 years and he'll be able to understand very well.
if you give a human the weights of qwen 0.8B and 5 years it'll still be impossible to edit the weights in any legible manner.
Is that so? There is quite a lot of research going on to identify the "meaning" behind certain directions. I don't see why a human with enough time, perseverance and ability to run experiments with the model shouldn't be able to uncover some of the underlying mechanisms of the model.
Only of course not with the speed with which an LLM can do it.
I think the presentation of the points is confusing people, the model could understand weights directly for extremely small networks in the 10s of thousands of weights.
Once they moved onto even the smallest LLMs, what it did was closer to abliteration: create examples where the model does something you don't want, create examples where the model it doesn't do the thing, then most importantly do forward passes.
Once you're allowed to actually do a forward pass vs just reading raw weights, tracking the changes in activations and tweak the model to modify them is a very well established process that takes a few hundred lines of Python.
mmm, maybe they are?
John Lilly's long awaited English ⇔ Dolphin translator, my beloved!
IMO, this is a similar behavior that we saw from AlphaFold, which also used a type of Transformer architecture to make its protein predictions.
function policy(x){for(var i=100,p,b,t,s,c,r,a,h,M=Math.tanh,R=v=>v<0?-1-v:v>9?19-v:v;i--;)x[i*6]&&(p=i%10),x[i*6+3]&&(b=i),x[i*6+4]&&(t=i);r=b/10|0;c=b%10;s=c>t%10||c==t%10&&c<9?1:-1;a=R(c+s*(8-r));t=t/10|0;h=1e3*M(.02*((r>t||r==t&&r-8?r<6?a:(a+R(c+s*(9-r)))/2:R(c+s*(r+9-2*(r<2?0:r<4?r:4))))-p));return[0,30*M(-h-15),-100,30\*M(h-15),-100,-100]}
Maybe the models have a common internal representation for computations, and they're capable of consuming and producing computations in the syntax of neural network weights?
They can triangulate efficiently on the meaning from all the text at once.
It is why they are so good at processing ambiguity.
They can absorb an enormous amount of text and correlate the last thing that was said directly the first thing. That is much more like "seeing" than reading, or any other linear process - even though the field of vision is linear.
Lady in red dress scene from Matrix comes to mind.
Our visual system deals with TB of video input every second but we are able to filter through it and spot the important bits. We can even with closed eyes make our mind spin up visual scenes and alter them. I think that's quite possibly a useful explanation of what's happening here.
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.