Show HN: Coder Eval – A Framework for Evals

Posted by mihaich on Friday September 04, 2026 @03:09PM
from the show-hn dept.

We needed a framework to write our evals in and easily update them, run A/B tests, set all kinds of constraints (ex: timeouts, number of turns), and configure the execution environment (ex: sandboxes, dependencies, how to handle AskQuestion). We also have an agent judge.

It’s all in a YAML file now.

We seem to be moving toward a world where companies rely more and more on evals to decide what to ship, so this should be super helpful for normalizing evals across teams.

( 2 comments | 2 points | on HN )

Re:Show HN: Coder Eval – A Framework for Evals by paguasmar on Friday September 04, 2026 @04:17PM (#49569715)

Within 10 secs of reading this I was convinced I needed this in our pipeline.

For me the next step is plugging this into the company CI/CD to gate SKILL updates and catch tool call regressions before merging

[ Parent | on HN ]

Re:Show HN: Coder Eval – A Framework for Evals by tom3141 on Monday September 07, 2026 @01:56AM (#49594393)

finally a framework that allows me to monitor my coding agents!

[ Parent | on HN ]