News for hackers. Stuff that matters.
Re:Task-Specific LLM Evals That Do and Don't Work by Jerrrry on Friday December 13, 2024 @11:19AM (#42409886)
It may be the self-aware human bias tainting this reasoning, but it seems convergent with our own psyche/brain processes, and/or inherent to the way we commonly express conscious thoughts.

Percolating tokens that allow a more "accurate" latent space appear to be more accurate, but are nearly actually useless noise. Almost a virtual shower thought.

Because people only put the answer at the end of a grammatically correct statement, with the more "reasoned" statements being more articulately percolated/logically sound, and that is expressed grammatically. These statements are inferred to be associated with intellectual boiler-plate. They may be correlated and not actually causative, but that would require a multiple component architecture with embeddings being used as a proto-"qualia" and that is getting hairy.

Facts should "only" have to be read once, and should be explicitly defined with a more secure of a confidence completely. Implicit inferences from those explicit facts should be emitted from a different, less confident module; with the chat boilerplate being tacitly composed finally when presenting the output to the user.

Of course separating the baby from the bathwater is the hard (not impossible) part.

There are no comments on this comment yet.