Re:Internal representations of LLMs encode information about...
by genrilz on Thursday October 31, 2024 @11:02AM
(#42007631)
To me the claims in the article read something like "we have found a way to identify execution paths in some common compiler architecture (which are the transformer architecture in the case of LLMs) which are often but not always associated with buggy code". This seems like a reasonable claim to make.
Re:Internal representations of LLMs encode information about...
by genrilz on Thursday October 31, 2024 @11:14AM
(#42007754)
Additionally, I think you may or may not be suspecting research malpractice. Obviously I don't have insider knowledge, but I would note that the idea of training probes in the middle layer of the model wasn't their idea. This paper cites other papers that already did exactly that. The contribution of this paper is simply that focusing on the middle layers for certain "critical tokens" gives a better signal than just checking the middle layers on every token.
It's of course possible that this paper in particular is fraudulent, but note that there is a field of research making the same basic claim as this paper, so this isn't some one off thing. A reasonable amount of people from different institutions would need to be in on it for the entire field to be fraudulent.
It's a panacea