More and more it would appear the failure modes are a training data issue
Of course you do. Just because you don’t see the LLM trying the ls command for the billionth time with the wrong arguments, just to be corrected by the harness, doesn’t mean it doesn’t happen. Of course I use the ls cli as an example. Could be any task. That’s what I mentioned about the harness and that’s why they shine in the fields with hard verification mechanisms and feedback loops.
(this has been my experience since ~DSv4 from April)
It’s not the harness that attempts to fix the broken tool calls. It’s the LLM.
> last time I saw a tool call loop
We are not talking about tool call loops
Then what did "Just to be corrected by the harness" mean?
> last time I saw a tool call loop
Reasoning loops, output loops, etc, I just don't tend to see them with the models I use.
All I'm saying is that I find LLMs to have been consistently getting more reliable, to the point that their failure modes don't really stand out.
The failure modes of the LLM are still the same. Nothing has fundamentally changed, despite the UI and the harness showing significant improvements and outcomes.