Re:In 27,280 coding agent runs, 64% of inference spend went ...
by scosman on Tuesday November 17, 2026 @02:41PM
(#50537055)
We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.