News for hackers. Stuff that matters.
Re:Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA by gertlabs on Wednesday July 22, 2026 @01:43AM (#49002302)
Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other.

We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.

Data at https://gertlabs.com/rankings

Re:Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA by isityettime on Thursday July 23, 2026 @10:58AM (#49022716)
Although you mention cost, I don't see task cost or total evaluation cost per model in your data. Am I just missing it?
Re:Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA by gertlabs on Thursday July 23, 2026 @01:05PM (#49024871)
The Efficiency tab at https://gertlabs.com/rankings?mode=oneshot_coding (only have cost data for the coding evaluations)
Re:Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA by isityettime on Thursday July 23, 2026 @02:53PM (#49026377)
Thanks!