Re:Making better decisions with the Brier score
by kqr on Monday June 07, 2021 @04:24AM
(#27420295)
I'm not (yet) using a scoring rule for my work-in-progress uncertainty test[1] of calibration, but only Beta posteriors, which are also a neat way of presenting the result of many predictions.
I am slightly more fond of log scoring than the Brier score, though, for the reason mentioned in another comment: being somewhat wrong is often worse than being very right, and should be penalised harder numerically.
(By the way, I build this to practise myself -- but I ran into a problem: I know the answers to all propositions, having written them myself... if anyone wants to contribute propositions, please contact me and I'll ask for them in a specific format so I can blindly paste them without knowing the true ones.)
I am slightly more fond of log scoring than the Brier score, though, for the reason mentioned in another comment: being somewhat wrong is often worse than being very right, and should be penalised harder numerically.
[1]: https://static.loop54.com/uncertainty-test.html
(By the way, I build this to practise myself -- but I ran into a problem: I know the answers to all propositions, having written them myself... if anyone wants to contribute propositions, please contact me and I'll ask for them in a specific format so I can blindly paste them without knowing the true ones.)