Language models can explain neurons in language models

Re:Language models can explain neurons in language models by Imnimo on Tuesday May 09, 2023 @03:19PM (#35879133)

To me the value here is not that GPT4 has some special insight into explaining the behavior of GPT2 neurons (they say it's comparable to "human contractors" - but human performance on this task is also quite poor). The value is that you can just run this on every neuron if you're willing to spend the compute, and having a very fuzzy, flawed map of every neuron in a model is still pretty useful as a research tool.

But I would be very cautious about drawing conclusions from any individual neuron explanation generated in this way - even if it looks plausible by visual inspection of a few attention maps.

[ Parent | on HN ]

Re:Language models can explain neurons in language models by mcguire on Tuesday May 09, 2023 @04:04PM (#35879680)

They also mention they got a score above 0.8 for 1000 neurons out of GPT2 (which has 1.5B (?)).

[ Parent | on HN ]

Re:Language models can explain neurons in language models by oofsa on Tuesday May 09, 2023 @05:45PM (#35880797)

I thought they had only applied the technique to 307,200 neurons. 1,000 / 307,200 = 0.33% is still low, but considering that not all neurons would be useful since they are initialized randomly, it's not too bad.

[ Parent | on HN ]

Re:Language models can explain neurons in language models by sebzim4500 on Tuesday May 09, 2023 @07:54PM (#35881891)

1.5B parameters, only 300k neurons. The number of connections is roughly quadratic with the number of neurons.

[ Parent | on HN ]