News for hackers. Stuff that matters.
Re:Show HN: Klarity – Open-source tool to analyze uncertaint... by siliconc0w on Monday February 03, 2025 @12:46PM (#42920789)
It seems like it would be easy to upgrade existing benchmarks to include uncertainty as a dimension. Then if a model is less certain it could maybe spend more time reasoning or route to a bigger model.
Re:Show HN: Klarity – Open-source tool to analyze uncertaint... by mrciffa on Monday February 03, 2025 @02:34PM (#42921883)
Exactly! Uncertainty is critical to correctly evaluate LLM performance and we don't need reasoning models to spend thousands of tokens on simple questions