News for hackers. Stuff that matters.
Re:AI benchmarks are a bad joke – and LLM makers are the one... by jiggawatts on Saturday November 08, 2025 @04:41PM (#45860321)
“Here’s the throughout at sustained 100% load with the same ten sample queries repeated over and over.”

“The customers want lower latency at 30% load for unique queries.”

“Err… we can scale up for more throughput!”

ಠ_ಠ

Re:AI benchmarks are a bad joke – and LLM makers are the one... by EdwardDiego on Sunday November 09, 2025 @02:18AM (#45863595)
And then when you ask if they disabled the query result cache before running their benchmarking, they blink and look confused.
Re:AI benchmarks are a bad joke – and LLM makers are the one... by nopurpose on Sunday November 09, 2025 @03:02AM (#45863793)
Then you see 25% cache hit rate in production and realise that disabling it for benchmark is not a good option either.