Re:Show HN: Factorio Learning Environment – Agents Build Fac...
by noddybear on Wednesday March 12, 2025 @09:28AM
(#43343033)
This is true - there are simpler benchmarks that can saturate planning for these models. We were motivated to create a broader spectrum eval, to test multiple capabilities at once and remain viable into the future.
[ Parent
| on HN ]
Re:Show HN: Factorio Learning Environment – Agents Build Fac...
by noosphr on Wednesday March 12, 2025 @03:43PM
(#43346931)
That's fair enough, but you should test other frontier model types to see if the benchmark makes sense for them.
For example the shortest path benchmark is largely usless when you look at reasoning models - since they have the equivalent of scratch paper to work through their answers the limitation became their context length rather than any innate ability to reason.