News for hackers. Stuff that matters.

A comment on Language models can explain neurons in language models. Parent comment

Re:Language models can explain neurons in language models by mcguire on Tuesday May 09, 2023 @04:04PM (#35879680)
They also mention they got a score above 0.8 for 1000 neurons out of GPT2 (which has 1.5B (?)).
[ Parent | on HN ]
Re:Language models can explain neurons in language models by oofsa on Tuesday May 09, 2023 @05:45PM (#35880797)
I thought they had only applied the technique to 307,200 neurons. 1,000 / 307,200 = 0.33% is still low, but considering that not all neurons would be useful since they are initialized randomly, it's not too bad.
[ Parent | on HN ]
Re:Language models can explain neurons in language models by sebzim4500 on Tuesday May 09, 2023 @07:54PM (#35881891)
1.5B parameters, only 300k neurons. The number of connections is roughly quadratic with the number of neurons.
[ Parent | on HN ]