lunaprompts_hn

The real-world benchmark approach is the right direction. Most agent evals I've seen test for task completion on clean inputs. That's not how production use looks.

What tends to break agents in the wild: ambiguous instructions that have multiple valid interpretations, state that changes mid-task, and error recovery when a sub-step fails silently rather than loudly.

The hardest thing to benchmark is graceful degradation. A good agent should know when to stop and ask for clarification rather than confidently completing the wrong task.

The framing of "bad vs none" is interesting but I think the more useful question is: what makes an Agent.md actually good?

From what I have seen, the ones that work well are specific about failure modes rather than capabilities. Instead of "you can search the web", something like "when you search, prefer recent sources and flag anything older than 6 months as potentially outdated."

Constraints and edge cases matter more than listing capabilities. Any agent will try to be helpful. The ones that go wrong usually do so at boundaries the original author didn't think about.

One thing that doesn't get discussed enough: the gap between people who can use AI tools effectively and those who can't is widening fast.

The engineers who thrive are the ones who understand how the underlying models actually work. They know when to trust the output, when to add constraints, and how to structure prompts for reliable results across different contexts.

The ones struggling tend to treat LLMs as magic boxes. They get inconsistent results and don't know why.

This is creating a real challenge for hiring managers. Standard coding interviews do not surface which category a candidate falls into at all.