Local Qwen isn't a worse Opus, it's a different tool

Re:Local Qwen isn't a worse Opus, it's a different tool by theshrike79 on Thursday June 18, 2026 @02:37AM (#48581581)

We shouldn't just measure the power of the raw LLM, harnesses matter more and more.

It's like taking the engine out a each car, putting it to a test bed and running it and then making a decision whether the car is good or bad based on the graphs the test bed provided.

You might have the best engine in the world, but if you put it in a shit car, the result is still bad. The seats are squeaky plastic, the infotainment is touch-only and you can't put on your seatbelt without knocking down whatever is in the cupholder.

[ Parent | on HN ]

Re:Local Qwen isn't a worse Opus, it's a different tool by sanderjd on Thursday June 18, 2026 @09:56AM (#48585429)

Aren't there benchmarks that measure at the harness level as well?

[ Parent | on HN ]

Re:Local Qwen isn't a worse Opus, it's a different tool by theshrike79 on Thursday June 18, 2026 @02:58PM (#48589859)

How would you benchmark "agent harness communicates with user clearly" it's 100% a feels measurement.

[ Parent | on HN ]

Re:Local Qwen isn't a worse Opus, it's a different tool by sanderjd on Thursday June 18, 2026 @08:59PM (#48593685)

I mean, in my experience some of this stuff is way closer to table stakes things than that. Like "the tool call didn't get totally confused" more than "did the communication with the user feel good".

[ Parent | on HN ]