News for hackers. Stuff that matters.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by xyzzy_plugh on Tuesday November 17, 2026 @12:52PM (#50535332)
I don't like this framing.

For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.

Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.

This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.

But I can't build a mobile app with this harness, or make a movie.

These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.

Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.

Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @01:10PM (#50535646)
I am thinking about how to measure the efficiency of ultilzing existing context in one specific domain.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by botanrice on Tuesday November 17, 2026 @01:19PM (#50535790)
hey! would you be so kind as to share your method or point me in the direction towards resources on how to setup a harness as you describe?

I've been tinkering with harnesses recently but have mostly used github copilot on vscode mostly thus far as my ai tool and feel like the harness setup on there is lacking (or really confusing...). many thanks if so :)

just trying to learn without googling "how to setup a harness" and get bombarded by a trillion yt vids or competing harness providers lol

Re:In 27,280 coding agent runs, 64% of inference spend went ... by xyzzy_plugh on Tuesday November 17, 2026 @02:58PM (#50537247)
Just ask one to write the other.

"Write me a harness in $LANGUAGE that uses the $MODEL API and has tools for X Y and Z" etc etc

I added remote management and web console over tailscale in about 20 minutes last week.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by NichoPaolucci on Tuesday November 17, 2026 @05:24PM (#50539118)
I would also consider just grabbing pi and tailoring it to your needs. It's super direct, no batteries, and I added a handful of smaller tools that blow everything else out of the water.

I've got some people on my team using 80-100 skills / plugins and their setup takes FOREVER to run even a simple modification.

Mine is just as correct and the feedback loop is lightyears faster, which is what I'm personally after.