News for hackers. Stuff that matters.
Re:AI leaderboards are no longer useful. It's time to switch... by nicklecompte on Tuesday April 30, 2024 @04:10PM (#40215717)
To be clear there’s a distinction between the quality of the results and the determinism of the results. If a low-precision LLM is wildly stochastic but the variation is mostly linguistic rather than factual or deductive (e.g. coin tosses on synonyms or presenting independent facts in a different order), then there’s not really a contradiction.

AFAIK the determinism side of floating-point precision hasn’t been well-addressed, but it’s been a while since I skimmed those papers.

There are no comments on this comment yet.