News for hackers. Stuff that matters.
Re:RTK reports token savings, but our cost benchmarks disagree by aeneas_ory on Friday September 11, 2026 @09:46AM (#49658357)
I literally say you should take benchmarks with a grain of salt :)

> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).

Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.

There are no comments on this comment yet.