One could say they might be "superhuman" here in the same way that LLMs might be said to be superhuman at math. Perhaps they do this work faster and more effectively than any person could manually. But they are certainly still doing it the "human" way ie reasoning with legible and coherent thoughts, rather communing through some Matrix-like symbology which is what people who skimmed the article seem to be assuming.

That "faster and more effectively" part is undeniably superhuman, and it can result in qualitative leaps in capabilities.

For example, an LLM's ability to comprehend and explain a complex codebase, that it's never seen before, far outstrips that of any human.

People like to come up with factors to quantify how much more efficient LLMs are. A newly hired human developer working on a complex codebase can take months to get up to speed, to get a holistic understanding of how it works so that they're able to make non-trivial changes. An LLM can do it in minutes. That's a factor on the order of 10,000x.

It may not be "communing through some Matrix-like symbology," but at some point it starts to seem that way.

Discussion on source:

https://news.ycombinator.com/item?id=50341174

"I think a smart human ML researcher could perform all of these tasks."

And do that for how long?

That's not really the point of the article IMO. It's more about how there's nothing magical that happens when my calculator does 347987987 x 55998. I could do it by hand too if I wanted, even if I wouldn't be nearly as fast. Yes, the calculator can do it much faster, for much longer and much more reliably, but there's nothing surprising about that.

The initial reporting/discussion surrounding these "LLMs understand LLM weights" made it sound as if these models had a superhuman ability to natively read and understand weights. Something that immediately seemed wild because I don't think that LLMs are trained to do that, so it would be an emergent ability.

If you look at the details though, it's fairly standard reverse-engineering: identify unique traits of the machine code you're looking at and make educated guesses as to what the code does, then use this understanding to unravel the next bit of code etc...

always one in the room

"If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games" -- absolutely not. Pac-Man in particular seems like a huge stretch to me.

> the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally.

This is why the distinction between open weights and open source is important.