Re:What happens when you analyze your favorite college footb...
by adam on Friday September 25, 2026 @09:10PM
(#49852126)
Not exactly - all the forecast questions get scored so we know how accurate the AI is and how calibrated it is. It's no different than asking a large crowd of humans on Good Judgment Open or Metaculus these same types of questions. Everything we ask eventually happens or it doesn't, and we can score it all.
Re:What happens when you analyze your favorite college footb...
by janalsncm on Friday September 25, 2026 @09:55PM
(#49852415)
Ok, so in an ML system you can calculate things like cross entropy loss, which penalizes your model for making confident, inaccurate predictions.
It doesn’t care whether your system is made of LLMs, decision trees, or bananas.
However, the problem is that unlike something like a neural net, you can’t exactly use backprop to improve.
On the flip side if the quality of the prediction doesn’t matter, I might as well have Claude spin up something shiny that does the same thing faster, cheaper, and with 10x the magic sparkles.
Re:What happens when you analyze your favorite college footb...
by what on Friday September 25, 2026 @09:45PM
(#49852358)
Okay. So how accurate is it?
Re:What happens when you analyze your favorite college footb...
by jschveibinz on Saturday September 26, 2026 @11:29AM
(#49857484)
I think it's important to to point out that there's a lot of "slop" to go around. I mean, making bad decisions and/or projections based on flawed data is nothing new.
The entire human scientific community has been shown recently on average to be generating "slop" and calling it science.
My bottom line: humans generate a lot of BS--and LLM's are just averaging it and giving it back to us.
It didn’t “calculate probabilities”, it just hallucinated slop. The same kind of slop almost started WW3 when a similar “intelligence analyst” system told the US navy to attack China because they were smuggling nukes into Iran. They weren’t, it was all slop. But their system was full of “calculated probabilities” too!