I think the developers explained the reason for this in a Reddit thread: Collectively a bunch of 1100 players are stronger than 1100. Imagine that a 1100 player will play one bad move for every two decent moves. However, different players miss different moves so the most picked move in each position will usually be a decent move.

This makes perfect sense - but is a bit problematic given the intended goal of the project. They've built a bot that plays like an 'averaged group' of humans, not a human. Maybe they need a better way of sampling from the outputs.

This is always the problem with training from historical data only: you’ll become very good at being just as good as the sample group.

Ideally you just use the sample as the basis, and then let an AI engine play against itself for training, and/or participate in real world games, such as they did with AlphaGo and/or AlphaStar.

Seems analogous to the average faces photography project where the composite faces of a large number of men or women end up being more attractive than you'd imagine for an average person.

This only true if you select the most likely move instead of sampling from the probability distribution over possible moves. In the latter case there is no reason for there to be a wisdom of the crowd effect.

If you sample from the probability distribution you are modeling, there is no reason it shouldn't play like a 1100 player.

Well one reason is that it still won't be a 'single' player - what they've done here is like having a group of thousands of 1100 players vote on a move. What you're suggesting is to then pick a random player each move and go with them. There's no consistency, maybe on move 10 the player is blind to an attacking idea, but then on move 11 suddenly finds it...

I think they are saying, if your neural network was probabilistic and you thought there was a 90% chance of someone doing move A, but a 10% chance of move B, then you shouldn’t always get move A if it was human like - you would sometimes get move B.

I.e. most of the time if you leave your queen hanging and under threat your opponent will take it, but sometimes they just don’t see it. That’s the difference between playing a bot and a human a lot of the time - humans can get away with a serious blunder more often at low level play.

That's exactly what I'm saying - except more like the model is saying there's a 90% chance that a randomly chosen player at this level would make the move.

Yes, if you don't condition on the past moves then the distribution you're modeling is where you randomly pick a 1100 player to choose each move as you say. What I'm saying is that there will be no wisdom of the crowd effect.

> maybe on move 10 the player is blind to an attacking idea, but then on move 11 suddenly finds it...

What part of that is unrealistic? This happens constantly to human players, and not only at 1100...

Unintentional ensemble learning :)

Maybe. I think if you had asked me to predict the rating I would have guessed below 1100 though. Because of only predicting moves in isolation. I would expect that the moves would not form a coherent whole working together in a good way.

So it's an interesting result to me.

That's what a few machine learning people I talked too thought would happen. There are lots of examples (self driving cars being the big one) in machine learning where training on individual examples isn't enough. We were actually hoping for our models to be as strong as the humans they're trained on so we are underperforming our target in that way.

I'm the main developer for Maia chess

Have you thought about trying a GAN or an actor-critic approach? Did you find it infeasible?

> Because of only predicting moves in isolation.

This actually may be the reason of higher ranking. What are the odds that a low ranked player will blunder a piece in a particular position? Quite low. But what are the odds that a low ranked player will blunder a piece in a game? They are quite high. So while this engine may predict most likely move, it can’t fake a likely game because it is too consistent.

I think GANs can be helpful to do something like this. One NN tries to make a man-like move and another one tries to guess whether the move was made by a human or the engine given the history of moves in a game.

I suppose the game at 1100 is really bad, such that it's mainly about avoiding obvious blunders, and not about having a sound long term strategy.

The real reason is that 1100 players are ranked ~1600 on Lichess. There's a good site that compares FIDE ratings, Lichess ratings, and Chess.com ratings. Lichess is inflated by many hundred points on the low end. Chess.com is more accurate. They converge towards the upper end of the human rating range.

I think the reason is that if you pick the most likely move for a 1100 player on every move, they would be a 1600 player.

Let’s say you accidentally leave a pawn hanging and 90% of 1100 players would spot it, and 10% of the time they miss it. In this engine, as the 90% is the most likely move it spots it 100% of the time.

So this example shows that if you pick the most likely move for a 1100 player every move, you end up scoring better than a 1100 player. Of course it works the other way too on spotting brilliant moves that others miss, but I guess at 1100 level play there are more opportunities to mess up!

The training data is also from lichess so I don't think that is it.

From the README on Github:

> Note also, the models are also stronger than the rating they are trained on since they make the average move of a player at that rating.

Reference: https://github.com/CSSLab/maia-chess