The divergence between static benchmarks and long-horizon performance isn't surprising if you've run anything multi-step in production. Benchmarks are short, isolated, well-specified. Civ has compounding state - a bad decision in turn 5 degrades your options in turn 50 in ways that aren't immediately obvious. It's a more honest signal than most standard evals.

The $1,200/match cost is the real constraint. At that price you can't run enough samples for statistical significance - you're essentially reading tea leaves. How are you handling context window management across 200 turns? Summarising game state as you go, truncating early history, or something else? The token accumulation over a full game must be substantial.

Also curious about the 90s timeout logistics. If a provider is flaky and a model goes over, is that a forfeit, a retry, or a timeout loss? Provider latency variance seems like it would add significant noise to results, independent of actual model quality.