I’ve been dying for someone to make a Civilization AI.

It might not be too crazy of an idea - would love to see a model fine-tuned on sequences of moves.

The biggest limitation of video game AI currently is not theory, but hardware. Once home compute doubles a few more times, we’ll all be running GPT-4 locally and a competent Civilization AI starts to look realistic.

I think it's also a matter of "shape". Like, GPT4 solves one "shape" of problem, given tokens, predict the next token. That's all it does, that's the only problem it has to solve.

A Civilization AI would have many problem "shapes". What do I research? Where do I build my city, what buildings do I build, how do I move my units, what units do I build, what improvements do I build, when do I declare war, what trade deals do I accept, etc, etc. Each of those is fundamentally different, and you can maybe come up with a scheme to make them all into the same "shape", but then that ends up being harder to train. I would be interested to see a good solution to this problem.

You can constrain LLMs (like LLAMA) to only output certain tokens that match some schema (e.g. valid code syntax).

I don't see why you can't get a LLM to output something like "research tech332; build city3 building24".

I am 100% certain that the training of such an AI will result in winning a game without ever building a single city* and 1,000 other exploits before being nerfbatted enough to play a 'real' game.

(That doesn't mean I don't want to see the ridiculousness it comes up with!)

* https://www.youtube.com/watch?v=6CZEEvZqJC0

If you train the model purely based on win rate, sure. Fortunately, we can efficiently use RLHF to train a model to play in a human-like way and give entertaining matches.

I knew it, I knew it! It would be a Spiffing Brit video.

That guy is a genius at finding exploits in computer games. I don't know how he does it, I think you need to play a fair bit of each game before you find these little corners of the ruleset.

Idk maybe he uses some sort of fuzzer

But wouldn't this be amazing for the developer to fix a lot of edge cases/bugs?

Maybe, maybe not. The stochastic, black-box nature of the current wave of ML systems gives me a gut feeling that using them like this is more of a Monkey's Paw wish granter than useful tool without a lot of refinement first. Time will tell!

> I’ve been dying for someone to make a Civilization AI.

Would love to see someone to make an AI that can predict our economy, perhaps by modeling all the actors that participate in the economy using AI agents.

Tbh I don't think an AI for Civ would that impressive, my experience is that most of time you can get away with making locally optimal decisions I.e growing your economy and fighting weaker states. The problem with current civ AI is that their economies can be often structured nonsensically, but optimized economies is usually just the matter of stacking bonuses together into specialized production zones, which can be solved via conventional algorithms.

Maybe, but a lot of people would like better AIs in strategy video games, it only adds to the frustration when people say "it wouldn't be that impressive". It's like saying "that would be easy... but it's not going to happen." (And I'm not focused on Civilization, it's just a well known example, I'd like to see a strong AI in any similar strategy game.)

I think it might be harder than StarCraft or Dota. Civilization is all about slow decision making (no APM advantages for the AI), and all the decisions are quite different, and you have to make them in a competitive environment where an opponent can raid and capture your cities.

The problem with game AI is that they "cheat". They don't play like a human. The civ AI straight up gets extra resources, AlphaStar in SC2 performed inhuman feats like giving commands in two different areas of the map simultaneously or spiking actions per minute to inhuman levels briefly. But even with all of that the AI still eventually loses. And then they start losing consistently as players play more against them.

Why? Because AI doesn't learn on the fly. The AI does things a certain way and beating it becomes a puzzle game. It doesn't feel like playing against a human opponent (although AlphaStar in SC2 probably came pretty close).

Learning on the fly is probably the biggest thing that (game) AI is lacking in. I'm not sure there's an easy solution for it.

>> Usually I see some signs of learning, but it fails to come up with anything spectacular.

And even if it succeeds, it fails again as soon as you change the environment because RL doesn't generalise. At all. It's kind of shocking to be honest.

https://robertkirk.github.io/2022/01/17/generalisation-in-re...