When I say "alien" I don't mean "magic" and I don't mean "emergent new woo-woo scary ability". I mean, not human in speed/volume. No human can process that amount of raw bulk in that amount of time successfully. And indeed, it's not magic at all. It is enormously mundane actually. It reminds me of old computers being able to do arithmetic at insane speeds compared to humans. It's not magic at all, but to be honest, I have the distinct feeling of goalposts flying all around me. Being able to handle tasks of this complexity.. I don't know about you but I still have to adjust to computers being able to do this and that it's actually "not surprising". I guess I'm getting old.
I can see how you want to calibrate the enthusiasm I see around here as it can be a bit.. well, but it's not necessary for me. I'm not overly enthused by this because I don't care about the NN adjusting itself and RSI etc. I'm sure it's all quite basic but the fact it can do it at all is reason to be.. watchful, IMO. That is to say, I can appreciate the level of capability necessary to do this work successfully, but I do take your point seriously. It made me think, thanks.
But it's not what's happening here at all! Read the thing in detail and you'll see that it's not doing anything alien, it does exactly what you (or at least someone with at least some knowledge abouy neural nets) would do if you were asked to perform the task. It doesn't reason about weights in a magical way, instead it reads the easy to interpret parts or use directional ablation to tweak a model's behavior, exactly like how the ERP folks have been decensoring models to produce adult content over the past two years.
It's an interesting demonstration of ML knowledge and agentic ability, but it's not alien NY any means, and it's not actually surprising given how strong today's models are with agentic work.