News for hackers. Stuff that matters.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by scosman on Tuesday November 17, 2026 @02:41PM (#50537055)
We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.

I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.

There are no comments on this comment yet.