News for hackers. Stuff that matters.
Re:Goldman Sachs model to predict World Cup game results did... by jtolmar on Monday July 16, 2018 @03:43PM (#17544405)
Say I have two models - model A returns around 20% likelihood that the top team wins the world cup, and model B returns around 80% likelihood. I use both of the modeling techniques a few thousand times in various parallel universes, and both of them are exactly right - 20% of 20% predictions result in a win, and so on. Despite them both quantifying uncertainty accurately, isn't model B still better?
Re:Goldman Sachs model to predict World Cup game results did... by ghayes on Monday July 16, 2018 @03:58PM (#17544563)
Think about the actual uncertainty since anything can happen in the game itself. The easiest way to look at this is to play the game 100 times in a row (preferably in parallel universes, as you say). If team A wins in 60% of games, then that caps the ability to predict the result. You can predict a die roll to be 6 with an certainty of 17%. You can’t do any better.
Re:Goldman Sachs model to predict World Cup game results did... by jtolmar on Monday July 16, 2018 @06:36PM (#17545767)
Say I have a bag of dice one of each of the usual D&D denominations (d4, d6, d8, d10, d12, d20). I draw one at random, ask the models for predictions, and roll it. Model A ignores the information about which one I drew, and predicts a correct distribution of rolls (12.9% chance of rolling a 6). Model B correctly processes the information about which one I drew, and predicts a correct distribution given that information (I drew the d6 so 17% chance of rolling a 6). Both models give correct results overall, but Model B has higher probabilities on average, and I would say it is a better model.

A model should be judged both on how accurately it characterizes its uncertainty and how much evidence it's able to successfully make use of.

Re:Goldman Sachs model to predict World Cup game results did... by thaumasiotes on Monday July 16, 2018 @04:11PM (#17544664)
You can do better if you have foreknowledge or retroactive foreknowledge of the outcome of the die roll, which is the obvious suggestion of jtolmar's comment. If I know the recorded outcomes of a sequence of die rolls, I can have models that predict those outcomes to any accuracy I want. But they're not doing it by measuring the uncertainty involved in prospectively rolling the die.
Re:Goldman Sachs model to predict World Cup game results did... by thousandautumns on Monday July 16, 2018 @08:43PM (#17546412)
No, because if the underlying phenomenon happened 20% of the time, that’s what you want your model to predict. The point of the model is to describe reality as accurately as possible. So a model that predicts a particular outcome to happen 80% of the time, and the outcome actually does happen 80% of the time, isn’t any better or worse than a model that predicts an outcome to happen 20% of the time that happens 20% of the time.