News for hackers. Stuff that matters.
Re:A ragtag band of internet friends became the best at fore... by jemmyw on Tuesday March 12, 2024 @08:06PM (#39686436)
That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?
Re:A ragtag band of internet friends became the best at fore... by draaglom on Tuesday March 12, 2024 @08:20PM (#39686528)
Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots.

This is mine on Metaculus (mentioned in the article):

https://imgur.com/a/d3s67xk

There's still a bit of noise at 205 predictions but you can see a pattern emerging!

Re:A ragtag band of internet friends became the best at fore... by anonymous-panda on Tuesday March 12, 2024 @11:00PM (#39687663)
But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.
Re:A ragtag band of internet friends became the best at fore... by tunesmith on Tuesday March 12, 2024 @11:42PM (#39687888)
Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.
Re:A ragtag band of internet friends became the best at fore... by anonymous-panda on Wednesday March 13, 2024 @01:35AM (#39688391)
This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China.

Basically this is an exercise of garbage in and garbage out.

Re:A ragtag band of internet friends became the best at fore... by omeze on Wednesday March 13, 2024 @02:06AM (#39688522)
I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful
Re:A ragtag band of internet friends became the best at fore... by anonymous-panda on Wednesday March 13, 2024 @09:52AM (#39691218)
Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.
Re:A ragtag band of internet friends became the best at fore... by tunesmith on Wednesday March 13, 2024 @04:24PM (#39696977)
Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.
Re:A ragtag band of internet friends became the best at fore... by hackerlight on Tuesday March 12, 2024 @09:45PM (#39687194)
The article does dig into it. They have Brier scores.
Re:A ragtag band of internet friends became the best at fore... by jellyberg on Wednesday March 13, 2024 @08:31AM (#39690601)