News for hackers. Stuff that matters.
Re:The future of AI according to thousands of forecasters by Imnimo on Tuesday June 27, 2023 @04:16PM (#36498091)
Is there any already resolved question in which metaculus forecasters made a surprisingly accurate prediction about an AI advance?
Re:The future of AI according to thousands of forecasters by yunohn on Tuesday June 27, 2023 @04:28PM (#36498211)
Anyone can signup and vote, so I assume it tends towards the majority option tbh. Seems like obviously noisy data.
Re:The future of AI according to thousands of forecasters by tersh on Tuesday June 27, 2023 @04:46PM (#36498401)
The Metaculus prediction weights predictors by how accurate they've been.
Re:The future of AI according to thousands of forecasters by yunohn on Tuesday June 27, 2023 @04:56PM (#36498523)
Yes, but if voters are voting the majority (and probable) decision, that just continues the trend of predicting obvious outcomes.

It’s not something to do with understanding domain knowledge and voting based on scientific research.

Re:The future of AI according to thousands of forecasters by avionical on Tuesday June 27, 2023 @05:36PM (#36498998)
I wouldn't say AGI in 10 years is predicting an obvious outcome. And the same goes for many other questions on the site.
Re:The future of AI according to thousands of forecasters by yunohn on Thursday June 29, 2023 @01:40PM (#36523857)
It will trend towards the major opinion as it gets closer to 10yrs. They haven’t been around long enough.
Re:The future of AI according to thousands of forecasters by usaar333 on Tuesday June 27, 2023 @04:45PM (#36498397)
https://www.metaculus.com/questions/track-record/

Only 13 resolved binary questions where you had longer prediction horizons (1+ year), the accuracy is zilch in AI category - Brier score of 0.25 which is akin to just guessing out 50% for all questions. Generally overconfident.

(Other categories much better - Brier of 0.14 1 year out)

Re:The future of AI according to thousands of forecasters by avionical on Tuesday June 27, 2023 @04:58PM (#36498561)
There are several AI categories on the track record page; make sure you're not just selecting the one or you'll miss a lot. There's a careful analysis of the overall track record on AI questions here: https://www.metaculus.com/notebooks/16708/exploring-metaculu...

The short version is that the Brier score is much better than .25 for AI questions, and the weighted Metaculus Prediction is more accurate still.

Re:The future of AI according to thousands of forecasters by usaar333 on Tuesday June 27, 2023 @05:23PM (#36498853)
Good call on the categories.

> The short version is that the Brier score is much better than .25 for AI questions, and the weighted Metaculus Prediction is more accurate still.

Added more categories. 1 year out is 0.217. I agree that's better than chance, though "much better"?

That said, this is dominated by bad community predictions pre-2020 and there's not much data recently for binary questions. I agree that CRPS is better - but it's not clear to me from that link how early they are looking at questions - accuracy gets better closer to resolve date -- I'm claiming that longer-term predictions are shakier.

Re:The future of AI according to thousands of forecasters by avionical on Tuesday June 27, 2023 @05:53PM (#36499175)
And you may want to check the weighted Metaculus Prediction if you haven't already.
Re:The future of AI according to thousands of forecasters by Imnimo on Tuesday June 27, 2023 @06:21PM (#36499497)
Can I see the list of questions used in this analysis somewhere? Is it literally just the set of questions I see when I filter for "Resolved" and "Artificial Intelligence"?

My impression from browsing that set of questions is that it's a mix of pretty trivial things like "how expensive will chatGPT be?" or "when will Google release Bard?". There are very few questions in the bunch I'd even consider interesting, let alone ones where the metaculus prediction appears to have offered any meaningful insight.

Re:The future of AI according to thousands of forecasters by avionical on Tuesday June 27, 2023 @09:17PM (#36501054)
No, it's also 'AI and Machine Learning,' 'Artificial Intelligence, and 'Forecasting AI Progress.' The list of resolved questions is roughly this: https://www.metaculus.com/questions/?status=resolved&has_gro... (though that will include a few questions that have resolved since the analysis.)