News for hackers. Stuff that matters.
Re:A ragtag band of internet friends became the best at fore... by michaelt on Wednesday March 13, 2024 @06:05AM (#39689705)
That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction.

I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year.

I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government.

For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply.

For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?

Re:A ragtag band of internet friends became the best at fore... by yorwba on Wednesday March 13, 2024 @07:02AM (#39690049)
Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction?

If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.

Re:A ragtag band of internet friends became the best at fore... by harperlee on Wednesday March 13, 2024 @07:22AM (#39690158)
But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.