The short version is that the Brier score is much better than .25 for AI questions, and the weighted Metaculus Prediction is more accurate still.
> The short version is that the Brier score is much better than .25 for AI questions, and the weighted Metaculus Prediction is more accurate still.
Added more categories. 1 year out is 0.217. I agree that's better than chance, though "much better"?
That said, this is dominated by bad community predictions pre-2020 and there's not much data recently for binary questions. I agree that CRPS is better - but it's not clear to me from that link how early they are looking at questions - accuracy gets better closer to resolve date -- I'm claiming that longer-term predictions are shakier.
My impression from browsing that set of questions is that it's a mix of pretty trivial things like "how expensive will chatGPT be?" or "when will Google release Bard?". There are very few questions in the bunch I'd even consider interesting, let alone ones where the metaculus prediction appears to have offered any meaningful insight.
Only 13 resolved binary questions where you had longer prediction horizons (1+ year), the accuracy is zilch in AI category - Brier score of 0.25 which is akin to just guessing out 50% for all questions. Generally overconfident.
(Other categories much better - Brier of 0.14 1 year out)