News for hackers. Stuff that matters.
Re:A ragtag band of internet friends became the best at fore... by dadrian on Tuesday March 12, 2024 @08:03PM (#39686417)
You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.
Re:A ragtag band of internet friends became the best at fore... by jemmyw on Tuesday March 12, 2024 @08:06PM (#39686436)
That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?
Re:A ragtag band of internet friends became the best at fore... by draaglom on Tuesday March 12, 2024 @08:20PM (#39686528)
Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots.

This is mine on Metaculus (mentioned in the article):

https://imgur.com/a/d3s67xk

There's still a bit of noise at 205 predictions but you can see a pattern emerging!

Re:A ragtag band of internet friends became the best at fore... by anonymous-panda on Tuesday March 12, 2024 @11:00PM (#39687663)
But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.
Re:A ragtag band of internet friends became the best at fore... by tunesmith on Tuesday March 12, 2024 @11:42PM (#39687888)
Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.
Re:A ragtag band of internet friends became the best at fore... by anonymous-panda on Wednesday March 13, 2024 @01:35AM (#39688391)
This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China.

Basically this is an exercise of garbage in and garbage out.

Re:A ragtag band of internet friends became the best at fore... by omeze on Wednesday March 13, 2024 @02:06AM (#39688522)
I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful
Re:A ragtag band of internet friends became the best at fore... by anonymous-panda on Wednesday March 13, 2024 @09:52AM (#39691218)
Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.
Re:A ragtag band of internet friends became the best at fore... by tunesmith on Wednesday March 13, 2024 @04:24PM (#39696977)
Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.
Re:A ragtag band of internet friends became the best at fore... by hackerlight on Tuesday March 12, 2024 @09:45PM (#39687194)
The article does dig into it. They have Brier scores.
Re:A ragtag band of internet friends became the best at fore... by jellyberg on Wednesday March 13, 2024 @08:31AM (#39690601)
Re:A ragtag band of internet friends became the best at fore... by dotancohen on Wednesday March 13, 2024 @02:22AM (#39688578)
That sounds a bit like the gambler's fallacy to me.
Re:A ragtag band of internet friends became the best at fore... by avereveard on Wednesday March 13, 2024 @03:20AM (#39688825)
Doesn't seem right, if a state capital city base chance for a nuclear strike is 1% and you have 195 capitals or so 2 of them would be hit by now.
Re:A ragtag band of internet friends became the best at fore... by TOMDM on Wednesday March 13, 2024 @04:07AM (#39689078)
You're correct it mostly only works for a spread of predictions that don't have such a large influence on each other.

If one capital gets nuked the odds of another one suffering the same fate skyrocket.

Re:A ragtag band of internet friends became the best at fore... by dmurray on Wednesday March 13, 2024 @09:54AM (#39691234)
Yeah, it's just about possible to imagine Kyiv or Tehran gets nuked in a "limited engagement" that never escalates further. But if, say, Lisbon gets nuked then the whole world is at war.
Re:A ragtag band of internet friends became the best at fore... by danmaz74 on Wednesday March 13, 2024 @04:04AM (#39689066)
Nuclear strikes on N different capital cities aren't statistically independent events, and that kind of computation only makes sense for statistically independent events.
Re:A ragtag band of internet friends became the best at fore... by cudder on Wednesday March 13, 2024 @04:02AM (#39689059)
Maybe your base rate is way off then?
Re:A ragtag band of internet friends became the best at fore... by michaelt on Wednesday March 13, 2024 @06:05AM (#39689705)
That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction.

I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year.

I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government.

For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply.

For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?

Re:A ragtag band of internet friends became the best at fore... by yorwba on Wednesday March 13, 2024 @07:02AM (#39690049)
Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction?

If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.

Re:A ragtag band of internet friends became the best at fore... by harperlee on Wednesday March 13, 2024 @07:22AM (#39690158)
But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.