gopalraja

Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.

One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.

After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.

Yeah — i agree based on my own experience. It was that the loop only works because if you have a rubruc to measure againt. I’ve had similar issues but not in any decompiler mode. Agents can be deceptive in explaining something so well (and I was too lazy to carefully read it verify it myself). Where I got burned was when a beta tester pointed out some citations that didnt exist. So i put a check in place. agent can propose freely, unconfirmed stuff fails closed, my rubric checks get the veto. Same shape as what you’re describing. The agent can refine; the rubric metrics is what keeps it from just becoming fluent drift. Without that human in the loop, autonomous improvement is mostly relies on luck. On the other hand, I am seeing more and more the newer model's reasoning and having another llm eval (provided that llm has rubrics to measure against) does lessen the falseness.

Yeah, fair question.

I read this less as “build a decompiler with an LLM instead of training one to decompile,” and more as a different loop: you still get an artifact you can measure and improve against IDA / Ghidra / angr. Training on infinite compiler data is powerful for the first path; this post is mostly about the second one (i think, but i could wrong).