Re:Evaluating AGENTS.md: are they helpful for coding agents?
by regularfry on Tuesday February 17, 2026 @08:52AM
(#47047481)
> Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However, for example on Sonnet 4.5, performance _drops_ by over 2%. Qwen3 benefits most and GPT-5.2 improves by 1-2%.
Ok so that's interesting in itself. Apologies if you go into this in the paper, not had time to read it yet, but does this tell us something about the models themselves? Is there a benchmark lurking here? It feels like this is revealing something about the training, but I'm not sure exactly what.
Re:Evaluating AGENTS.md: are they helpful for coding agents?
by nielstron on Tuesday February 17, 2026 @12:26PM
(#47050160)
It could... but as pointed out by other the significance is unclear and per-model results have even less samples than the benchmark average. So: maybe :)
Ok so that's interesting in itself. Apologies if you go into this in the paper, not had time to read it yet, but does this tell us something about the models themselves? Is there a benchmark lurking here? It feels like this is revealing something about the training, but I'm not sure exactly what.