Re:Predicting outcomes for games of skill by redefining what...
by trendia on Tuesday February 06, 2018 @02:44PM
(#16318924)
There is an easy, quantifiable metric that uses this idea: the Brier score.
Essentially, if you make 100 predictions, the Brier score will measure how well you are able to make precise predictions. If you always estimate the odds that X will win to be 50/50, you won't do nearly as well as someone who correctly predicts the outcome each time.
Re:Predicting outcomes for games of skill by redefining what...
by morelandjs on Tuesday February 06, 2018 @05:06PM
(#16320354)
I didn't know about this, thanks. In the paper (long read I know), I use a modified quantile-quantile test to demonstrate that my predicted uncertainties are accurate.
For a single match, no. Across a sample of predictions? Elementary. (If 60% of the games you give white a 90% chance of winning are, in fact, won by black, you don’t need a lot of games to determine your model is wrong.)