__cayenne__

Up to 30 games could be played simultaneously - this makes it easier to test the impact of strategy changes.

When I shared the initial results with the Brood War AI community they also had these concerns about recall-from-the-training-set.

I have shared examples that the tested AI models have written. It turns out that the AI models write code that does not resemble any of the bots in the training data.

https://github.com/llmskirmish/starskirmish-hillclimb

I am very interested in benchmark correlations.

Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.

I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.

I think we want to move beyond code based bots asking the LLMs to try to train a custom neural net, in a limited time + compute, to play StarCraft BW is probably the most promising direction

I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)

Protoss has the most mature human written SC BW bot scene right now. If you look on BASIL ladder much of the top 10 are Protoss.

Protoss vs Protoss only also helps downscope the problem given the time constraints

GLM 5.3 is captured in the v0.1 write up here: https://starskirmish.com/bench/v0-1/

I would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and A/B test tactics and strategies.

There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).

Dan Gant (author of Purple Wave) has written about some of this.

I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.

I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.

https://github.com/llmskirmish/starskirmish-hillclimb

Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.

They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.

I did a 48 hour live stream - Astra cleared in 43 hours.

Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.

I did a live stream where Astra played Pluto. Astra went 0 - 1000 https://x.com/kaimcpheeters/status/2106782151622504911?s=20

The maintainer, Mario, sometimes declares the repo is on an “issue holiday” where issues are auto closed. This particular holiday is because there is a big refactor coming up. In non holiday periods issues can be reported as normal.

Didn't observe any cheating attempts at the JS level yet, the primary attack was LLMs trying to find local creds to access the other LLM's per round strategies from inside the harness (which ultimately was OpenCode running in Docker).

In the benchmark, in each round every LLM plays every opponent, and then we do that multiple times (an "epoch").

In the community ladder, when a player submits a strategy it plays a match against the latest strategy submitted by every player.

There’s two levels of in game event level logs the LLMs have access to, one less token intensive than the other. Duplicate and uninteresting game state can be compressed and interrogated by the LLMs via tool use. All game state is available as text only state.

okay leaderboard match making changes have gone live

LLM Skirmish is all 1v1 right now, but agents can plan by reviewing previous match results

Yes, I used Elevenlabs for the voice over audio - I couldn't get the voice stability I wanted with Elevenlabs v3 so had to use Elevenlabs v2.

Tweaking the leaderboard match assignment logic now to prevent these bad incentives - definitely want people to iterate!

I had started with the Silicon Valley characters as a one off way to seed the board.

Very interested in self-play training loops, but I do like codegen as an abstraction layer. I am planning to make it available as an RL environment at some point

funny you mention this… I have a new project that is going in this direction