News for hackers. Stuff that matters.

kiops

Ask HN: What eval harness holds up in practice, and what is still missing?

There’s no shortage of eval frameworks, but once you move past simple single-turn prompt checks into multi-step, tool-calling agents, most of them start to feel either too heavy or too brittle for CI. Testing intermediate trajectories, retries, and non-deterministic state often turns into flaky LLM judges, slow test runs, or snapshot diffs that become painful to maintain.For people evaluating...

( Read More... | 1 comment | on HN )
How does this handle harness-specific features without becoming a lowest-common-denominator API?
Owning the language, IDE, and engine might actually be the interesting part here. The question is what workflow that lets you build that would be awkward or impossible in Godot/GameMaker. If you find that, you probably find the project's killer feature too.