If the author (you?) had just described it in those terms, it'd be hard to object. But the article goes further, and makes claims about the system being more accurate due to a different rating curve. That's the claim that would need to be justified by actually comparing whether the predictions the new system makes really are better.
[0] http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-...
Like our tolerance for losing is acquired. Most normal people losing in League for the first time stop playing, usually forever. Just randomly visit your friend's match histories in League, frequent players have many days of long losing streaks.
If you're just conditioned to play despite losing, great, in a Darwinian way (surviving, being around to be measured) you will be representative of the average player in League. And there are so many League players with such long retention you cannot possibly argue that skill-based matchmaking is the core component of user engagement.
His dataset is interesting because it will necessarily overrepresent people who kept playing despite the old system. That sort of refutes its importance - I mean sure people complain but they keep playing, so was it really that important? So what if complaints go down?
Those are important goals, and also, it's still an interesting twist in multiplayer game design. You just gotta interpret it as a commentary on a whole system even if it doesn't narrowly talk about a scientific objective like performance prediction.
If somebody won against you with a particularly dirty strategy you can get back at him and you won't be fooled the next time.
It's better than dota system where you're most likely never going to play vs the same people again (or at least it feels like that).
Also the whole 5v5 game that lasts for 1 hour format is inherently very toxic - most of the time you know you lost after 10 minutes and yet you can't disconnect and have to keep playing for the next 30-50 minutes while everybody blames everybody else on your team.
In sc2 if you know you lost after 5 minutes it's discrespectful to continue playing :)
Sadly in a team game you can't make that choice because disconnecting too much will prevent you from playing all the fun modes (and will put you into a low priority queue where you play with people who constantly insult everybody or kill their own allies).
That's the root cause of all the toxicity in dota.
The chance that the enemy throws a fight or makes other mistakes that opens up new win conditions (or sometimes that simply one guy in the other team tilts and throws) is just too high in Dota in order to be able to call it quits after 10 minutes.
One of the main reasons for the toxicity in Dota in my opinion is that people don't understand that there are 10 variables in the field, but they can only control one of them. Don't even bother trying to influence the other 9 (apart from positive feedback/encouragement/communication to the team), it's a waste of time and only contributes to the toxicity.
At this level with the same position and firstpicking ww I get a small subset of heroes I play with and against, so it's much easier to call. I have pudge and sniper every other game for example, and enemy pos 5 is very often cm or ogre :) It's a great day I get to play vs meepo or oracle but it almost never happens. The best I can hope for is axe or legion.
Also half the fancy strats don't work because people don't talk. Split-pushing is just split-feeding. Nobody does 3-lanes. Jungling is a last resort of carry that lost the lane and when he's fat the barracks are already destroyed.
So if we lost laning hard enough and have earlier-game cores than the enemy - we lost, it just takes a lot of time to play out.
But even if I was wrong 30% of the time instead of 1o% - I'd still happily take that trade in mmr if I could. Obviously pro players have different motivation so for them it's "never surrender", but almost nobody is a pro.
More importantly, though, your MMR has now been split up across the three factions that you can play as. If your Terran gameplay is much better than your Zerg gameplay, you'll be facing higher-ranked opponents when you play Terran, than when you play Zerg.
This has made it easier for players to switch their factions up, without being punished for it.
That is, perhaps the system could be engineered to maintain a more even win/loss ratio so that people don't go on super-long win (or loss) streaks in general by adjusting who they get matched with.
It probably wouldn't work that well towards the edges, but around the middle it might work well enough.
On the other hand I suspect the historical data really is the "best fit" to the historical data.
Personally, if I was designing a rating system, I would use two separate systems.
One would be like the one in this game; publicly viewable, pleases the players, and gives a sense of accomplishment.
Then, I would have a second, internal only rating system that players can't see but is used for matchmaking to make sure people are matched up to players with as close to equivalent skill as possible.
Peple knew this and ignored xp altogether.
Naturally future data is much harder to deal with than past data. But even for future data it's not obvious that ELO (or any other theoretical fit to the odds of winning) will be more accurate than the historical odds.
You raise a good point in that I could've created a training set and a test set, that probably would be a better validation. But I don't know, I'm not doing science, I'm making a game.
On the topic of whether the future matches the past, the predictions were based on a rolling database of the past 100000 matches, which is approximately the number of matches played per 7 days. So my theory is that the data is quite recent and up-to-date and so should match, in general.
Of course I never tested this. In the end, I'm not doing science, I'm making a game. If the retention goes up, complaints are down, then I can't keep working on the rating system, there are 1000 other things to do.
By gum, an opportunity to quibble semantics on the internet. That is true if benchmark using means 'only admit to knowing' and accuracy means 'must be numerically quantifiable given existing data'. It is false otherwise, especially if accuracy means 'conforming to truth' and we have a model for how the numbers are being generated.
Obviously if I generate a set of numbers by sampling a normal distribution then the most accurate model is a normal distribution, no matter what empirical data I use for benchmarking.
That is to say, if we know how the data was generated (sans noise) we can reject empirical distributions as the most accurate, because we can directly know the distribution of the data.
Generally speaking the "rating structure" is a lattice where you can, for any two players A and B, tell whether A is a better player than B or the other way around. Elo, Glicko, etc. are embeddings of this lattice on the real line (much like the utility functions of microeconomics are real embeddings of preference lattices).
Not really. You tend to have cycles, where person A can beat person B who beats person C who beats person A.
There's a guy who was even with the guys playing Go who were 2 or 3 stones weaker than me that would tend to beat me because of some of the unorthodox things he did. (Eventually I strengthened my game against these things).
Considering ratings to be a total ordering is a useful approximation.
Others have pointed out how there is a psychological aspect of rating systems, and no developer wants to constantly field complaints. That said, I believe the answer is yes. A rating system derives meaningfulness from its predictive power. In other words, people want to know how good they actually are compared to one another.
There's an interesting question about how you rate players who use fundamentally different strategies. For instance in RTS games, should you match boom vs blitz players of otherwise equivalent elo? Or should you instead (try to) construct a classifier to determine which type of player someone is, and then have a rank against each other type of player and match them according to that rank?
That's why people use barcode names (llll1l1l1l1l111l) in starcraft :)
At high skill levels players are skilled enough to where it might, but with most online multiplayers the overwhelmingly vast majority of players are lacking in basic fundamentals to varying degrees. At that level, ELO based matchmaking mostly just results in one person getting rolled or doing the rolling. They’re not really competitive games in my experience.
Some of these complaints are solved by existing systems, namely Glicko. For example, rating deviation helps with experienced players (low RD) losing points to newer players (high RD). It also has a built-in way to discourage inactivity. Players' RD increase over periods of inactivity, so they can be excluded from the leaderboard after reaching a certain point. That allows us to maintain their rating without decreasing it. After all, that's our best guess of the player's skill. It's just a less reliable guess over time.