Is that so? There is quite a lot of research going on to identify the "meaning" behind certain directions. I don't see why a human with enough time, perseverance and ability to run experiments with the model shouldn't be able to uncover some of the underlying mechanisms of the model.
Only of course not with the speed with which an LLM can do it.
I think the presentation of the points is confusing people, the model could understand weights directly for extremely small networks in the 10s of thousands of weights.
Once they moved onto even the smallest LLMs, what it did was closer to abliteration: create examples where the model does something you don't want, create examples where the model it doesn't do the thing, then most importantly do forward passes.
Once you're allowed to actually do a forward pass vs just reading raw weights, tracking the changes in activations and tweak the model to modify them is a very well established process that takes a few hundred lines of Python.
that is why this is crazy,
give a human a legacy codebase and 5 years and he'll be able to understand very well.
if you give a human the weights of qwen 0.8B and 5 years it'll still be impossible to edit the weights in any legible manner.