News for hackers. Stuff that matters.
Re:Predicting outcomes for games of skill by redefining what... by macromaniac on Tuesday February 06, 2018 @12:23AM (#16314596)
I like the way elo is computed. Win/loss is the most fair metric because it by definition doesn't overvalue any particular in game statistic, everything is weighted perfectly. Still good read though.
Re:Predicting outcomes for games of skill by redefining what... by cjslep on Tuesday February 06, 2018 @02:25AM (#16314896)
I prefer Glicko and Glicko-2 over Elo. It's been 10 years since I implemented it and played with the math, but if I remember right it is far more accurate in games that have significant amounts of RNG that leads to lucky wins/loss events. But if a player proves they weren't lucky by being consistent it still is just as malleable as Elo. So perfect for a game like Hearthstone.
Re:Predicting outcomes for games of skill by redefining what... by sukruh on Tuesday February 06, 2018 @07:27AM (#16315812)
Also relevant is TrueSkill[1] where the game is between teams of people who are individually ranked.

[1] http://trueskill.org

Re:Predicting outcomes for games of skill by redefining what... by highd on Tuesday February 06, 2018 @02:53AM (#16314959)
Regarding this and previous discussions on this topic on HN, it seems to me that one of the primary motivating factors when constructing a new ranking system should be the possibility of cyclical dominance a la rock/paper/scissors, which we should expect to see in many modern complex games. If one wanted to solve that problem, I think it would require some form of multidimensional ranking. Then one would have added flexibility so you could accurate predict winners from the scoring with some function f(x_1,x_2) even in cyclical cases. This would be less interpret-able, but it would have a fair shot at actually modelling/predicting/depicting the dominance relationships between players.
Re:Predicting outcomes for games of skill by redefining what... by morelandjs on Tuesday February 06, 2018 @05:04PM (#16320337)
I agree this is a critical feature that Elo ratings omit. Elo ratings are transitive, i.e. if I know the probability that A beats B and the probability that B beats C, then I know the probability that A beats C. It's almost certainly the case that non-linear "rock-paper-scissors" effects exist in professional sports.

The tricky bit is trying to estimate these effects with a finite number of samples or observations. Elo thrives on limited data.

Re:Predicting outcomes for games of skill by redefining what... by everdev on Tuesday February 06, 2018 @12:06AM (#16314545)
For me the most frustrating part of predicting outcomes (elections, games, etc.) is that unless you're predicting with 100% confidence (which I've never seen), that no one can essentially call your prediction wrong.

If a prediction model was consistently accurate, the creators would be breaking Vegas instead of publishing predictions on a blog.

Re:Predicting outcomes for games of skill by redefining what... by JumpCrisscross on Tuesday February 06, 2018 @01:46AM (#16314810)
> no one can essentially call your prediction wrong

For a single match, no. Across a sample of predictions? Elementary. (If 60% of the games you give white a 90% chance of winning are, in fact, won by black, you don’t need a lot of games to determine your model is wrong.)

Re:Predicting outcomes for games of skill by redefining what... by trendia on Tuesday February 06, 2018 @02:44PM (#16318924)
There is an easy, quantifiable metric that uses this idea: the Brier score.

Essentially, if you make 100 predictions, the Brier score will measure how well you are able to make precise predictions. If you always estimate the odds that X will win to be 50/50, you won't do nearly as well as someone who correctly predicts the outcome each time.

https://en.m.wikipedia.org/wiki/Brier_score

Re:Predicting outcomes for games of skill by redefining what... by morelandjs on Tuesday February 06, 2018 @05:06PM (#16320354)
I didn't know about this, thanks. In the paper (long read I know), I use a modified quantile-quantile test to demonstrate that my predicted uncertainties are accurate.

https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot

Re:Predicting outcomes for games of skill by redefining what... by klodolph on Tuesday February 06, 2018 @12:20AM (#16314587)
It's not relevant and not interesting that you can't call an individual prediction "wrong". The article is just presenting a way to extend Elo to point spread. So it's better than win-loss Elo, not better than more complicated statistical models, which the authors explicitly state:

> Our findings indicate the probabilistic predictions are highly accurate with a mean absolute error that is comparable to, albeit slightly worse than, Vegas betting lines.

Meanwhile, tons of people are throwing ML models at sports bets, models that they often don't understand, and Vegas is keeping up with it by adjusting the payouts.

Re:Predicting outcomes for games of skill by redefining what... by richard___ on Tuesday February 06, 2018 @12:42AM (#16314652)
No one's breaking vegas because most games and payouts are designed probabilistically for the house to win
Re:Predicting outcomes for games of skill by redefining what... by joosters on Tuesday February 06, 2018 @05:09AM (#16315341)
I think they are referring to sports betting rather than games of chance. There’s no point calculating an elo rating for number 5 on the roulette wheel.
Re:Predicting outcomes for games of skill by redefining what... by carbocation on Tuesday February 06, 2018 @01:21AM (#16314756)
Since Elo and PageRank can sometimes (but not always) be used in similar circumstances, is there a PageRank generalization that is similar to this approach for Elo?
Re:Predicting outcomes for games of skill by redefining what... by morelandjs on Tuesday February 06, 2018 @05:13PM (#16320411)
I'm going out on a limb here because I'm not familiar with page range, but yes you could use our margin-dependent Elo model to rank items.

Classic Elo: A user visits a page and is shown a picture of two different types of Halloween candy. The user clicks the candy they like better. You run "win/loss" Elo on the candy matchups and rank the candies by their Elo rating.

FiveThirtyEight did that here http://fivethirtyeight.com/features/the-ultimate-halloween-c...

Margin-dependent Elo: A user visits a page and is shown a picture of two different types of Halloween candy. The user moves a sliding bar, say from -10 to 10, to indicate which candy they prefer. I can then run margin-dependent Elo rankings on those candy matchups and rank the candies by different moments of their Elo rating distributions.

Re:Predicting outcomes for games of skill by redefining what... by thanatropism on Tuesday February 06, 2018 @07:57AM (#16315941)
PageRank would weigh all games the same -- unless graph weights are somehow adjusted. Elo naturally weighs down older games.

Put differently: PageRank models stable dominance relations, while Elo better reflects evolution in skill.

Re:Predicting outcomes for games of skill by redefining what... by carbocation on Tuesday February 06, 2018 @08:33AM (#16316098)
Succinct and clear. Thanks!
Re:Predicting outcomes for games of skill by redefining what... by pokoleo on Tuesday February 06, 2018 @01:05AM (#16314711)
Cleaner version of the paper is on arxiv-vanity:

https://www.arxiv-vanity.com/papers/1802.00527/

Re:Predicting outcomes for games of skill by redefining what... by occamrazor on Tuesday February 06, 2018 @02:46AM (#16314947)
I get an error page (on mobile)