News for hackers. Stuff that matters.
Re:Show HN: Factorio Learning Environment – Agents Build Fac... by p10jkle on Tuesday March 11, 2025 @08:15AM (#43331679)
Wow, fascinating. I wonder if in a few years every in-game opponent will just be an LLM with access to a game-controlling API like the one you've created.

Did you find there are particular types of tasks that the models struggle with? Or does difficulty mostly just scale with the number of items they need to place?

Re:Show HN: Factorio Learning Environment – Agents Build Fac... by noddybear on Tuesday March 11, 2025 @08:23AM (#43331727)
Hey - yes, I think this is definitely possible, as you don't need any training compute for it to work. Its super easy to plug-and-play different models into new games, once an API is made available.

Models struggle in 2 main areas. The first is spatial reasoning: often the models make off-by-one errors which they find it hard to recover from (as factories are very sensitive to these mistakes - like in programming). The second is in long-term planning, i.e figuring out what to do strategically, before making tactical subgoals.

The difficulty scales in lab-play generally in proportion to the depth of the production chains. If an item requires several factory segments first, this makes it a lot more challenging. I think this is related to planning though, as the models tend to get down 'into the weeds' of fixing minor issues - rather than coming up with a master plan first.

Re:Show HN: Factorio Learning Environment – Agents Build Fac... by pyinstallwoes on Tuesday March 11, 2025 @08:37AM (#43331824)
Have you tried specific prompting like writing a mermaid diagram that forces the model to contextual use long term horizon tasks ?
Re:Show HN: Factorio Learning Environment – Agents Build Fac... by noddybear on Tuesday March 11, 2025 @09:11AM (#43332075)
Yes we tried that - as well as a few other visual DSLs for spatial reasoning. They didn't seem to help much, i.e there were no failure modes that this approach solved compared to the simpler approach. As ARC-AGI results showed - there don't seem to be many 'free lunch' solutions to this without actually training.
Re:Show HN: Factorio Learning Environment – Agents Build Fac... by posterman on Tuesday March 11, 2025 @08:33AM (#43331794)
"claude plays pokemon" shows that it struggles with mount moon (as did four year old me)
Re:Show HN: Factorio Learning Environment – Agents Build Fac... by noirscape on Tuesday March 11, 2025 @09:23AM (#43332192)
Very unlikely that you'll see mass-use of LLMs as opponents. Enemy AI in most games doesn't need the level of complexity that machine learning demands. (Ignoring computational costs for a second.)

The main goal of an enemy AI isn't to be the hardest thing in the world, it's to provide an interesting challenge for the player to overcome. It's not necessarily difficult to make a hypercompetent AI in most games, but that also wouldn't make it very interesting to play against. Most games have finite states of logic, just large enough to the point where a human would have trouble finding every solution to it (although humans tend to be very good at pushing on the edges of these states to find ways around them).

Even in games where the amount of state is much higher than usual, you rarely want a super AI; nobody likes playing against an aimbot in an FPS for example.

Factorio is an outlier because unlike regular games, the true condition for a "victory" is almost entirely up to the player. You can make a rocket in non-DLC Factorio (the games victory condition) without building any factory at all beyond the most basic structures for stuff you can't handcraft. It'd be extremely slow, but it's an option. That's why the benchmark for this sort of thing is more efficiency than it is "can this work".

Re:Show HN: Factorio Learning Environment – Agents Build Fac... by PetitPrince on Tuesday March 11, 2025 @09:53AM (#43332471)
As an opponent that would be indeed unfun, but as a sparring partner / coach in a competitive game (fighting game? Rts? Moba? Puzzle game?) that would be useful.
Re:Show HN: Factorio Learning Environment – Agents Build Fac... by fragmede on Tuesday March 11, 2025 @03:16PM (#43336036)
Civilization (VII just released) is famous for having the harder difficulties be harder because the AI cheats. If the game was harder because the AI was smarter instead of it cheating, it would be worth it to players to upgrade!
Re:Show HN: Factorio Learning Environment – Agents Build Fac... by jkhdigital on Tuesday March 11, 2025 @10:57PM (#43339521)
Why LLM? Isn’t this what AlphaZero is good at? There are many more kinds of useful ML models than LLMs!