Re:Evaluating AGENTS.md: are they helpful for coding agents?
by viraptor on Tuesday February 17, 2026 @02:26AM
(#47044663)
You can also save time/tokens if you see that every request starts looking for the same information. You can front-load it.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by sebazzz on Tuesday February 17, 2026 @02:40AM
(#47044734)
Also take the randomness out of it. Sometimes the agent executing tests one way, sometimes the other way.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by Maxion on Tuesday February 17, 2026 @07:21AM
(#47046730)
I've found https://github.com/casey/just to be very very useful. Allows to bind common commands simple smaller commands that can be easily referenced. Good for humans too.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by NicoJuicy on Tuesday February 17, 2026 @03:27AM
(#47045006)
Don't forget to update it regularly then
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by averrous on Tuesday February 17, 2026 @01:47AM
(#47044436)
Agree. I also found out that rule discovery approach like this perform better. It is like teaching a student, they probably have already performed well on some task, if we feed in another extra rule that they already well verse at, it can hinder their creativity.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by imiric on Tuesday February 17, 2026 @02:04AM
(#47044541)
That's a sensible approach, but it still won't give you 100% confidence. These tools produce different output even when given the same context and prompt. You can't really be certain that the output difference is due to isolating any single variable.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by pamelafox on Tuesday February 17, 2026 @02:38AM
(#47044714)
So true! I've also setup automated evaluations using the GitHub Copilot SDK so that I can re-run the same prompt and measure results. I only use that when I want even more confidence, and typically when I want to more precisely compare models. I do find that the results have been fairly similar across runs for the same model/prompt/settings, even though we cannot set seed for most models/agents.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by ChrisGreenHeur on Tuesday February 17, 2026 @03:43AM
(#47045115)
same with people, no matter what info you give a person you cant be sure they will follow it the same every time
I do not do this for all repos, but I do it for the repos where I know that other developers will attempt very similar tasks, and I want them to be successful.