Re:Without benchmarking LLMs, you're likely overpaying 5-10x
by andy99 on Tuesday January 20, 2026 @04:52PM
(#46698225)
You obviously know what you’re looking for better than me, but personally I’d want to see a narrative that made sense before accepting that a smaller model somehow just performs better, even if the benchmarks say so. There may be such an explanation, it feels very dicey without one.
[ Parent
| on HN ]
Re:Without benchmarking LLMs, you're likely overpaying 5-10x
by jmathai on Wednesday January 21, 2026 @03:12PM
(#46710921)
Volume and statistical significance? I'm not sure what kind of narrative I would trust beyond the actual data.
It's the hard part of using LLMs and a mistake I think many people make. The only way to really understand or know is to have repeatable and consistent frameworks to validate your hypothesis (or in my case, have my hypothesis be proved wrong).
You can't get to 100% confidence with LLMs.
Re:Without benchmarking LLMs, you're likely overpaying 5-10x
by vercaemert on Friday January 23, 2026 @05:38AM
(#46730879)
You just need a robust benchmark. As long as you understand your benchmark, you can trust the results.
We have a hard OCR problem.
It's very easy to make high-confidence benchmarks for OCR problems (just type out the ground truth by hand), so it's easy to trust the benchmark. Think accuracy and token F1. I'm talking about highly complex OCR that requires a heavyweight model.
Scout (Meta), a very small/weak model, is outperforming Gemini Flash. This is highly unexpected and a huge cost savings.
Some problems aren't so easily benchmarked.