News for hackers. Stuff that matters.
Re:A group of 3,000 citizens is making better forecasts than... by jerf on Wednesday April 02, 2014 @09:24AM (#7515300)
"and the sample set is large enough, you would -expect- there to be some outliers who mostly got things right, by pure chance."

According to the article, "In fact, she's so good she's been put on a special team with other super forecasters whose predictions are reportedly 30 percent better than intelligence officers with access to actual classified information."

If they mean that after the team was formed, the predictions are still 30% better, than random chance ceases to be an acceptable explanation. If they mean that the team that they formed was 30% better on the predictions used to grant entrance into the team, then your objection still holds. Unfortunately, I don't think the article makes it clear which is the case.

That said, there are definitely some mechanisms that could lead an outside group to higher accuracy on these sorts of things. The two biggest are: 1. The insiders are simply that bad, for whatever reason, and random chance can easily beat them, and 2. The failures of the insiders are correlated and overwhelm their theoretically superior information, and I'd especially finger potential ideological biases there, which could affect both information filtering and the analysis. (It is one thing to believe the world should be a certain way as a result of one's ideology... it is quite another to believe the world must already be a certain way as a result of one's ideology, even when one is staring evidence in the face that it is not. One is noble and laudable... the other somewhat less so.)

Re:A group of 3,000 citizens is making better forecasts than... by Ovid on Wednesday April 02, 2014 @09:39AM (#7515403)
As a guess, I would suspect that the researchers who set up the experiment are more careful about their analysis than NPR is with its reporting. NPR needs to popularize its information, thus ensuring that we, the casual reader, can get a grasp of what's going on. That necessarily dumbs down the content.
Re:A group of 3,000 citizens is making better forecasts than... by vitoreiji on Wednesday April 02, 2014 @10:50AM (#7515905)
Why did they go half way, though? If you're going to dumb down things, why don't just say "scientists found that" instead of giving a distorted view of the experiment, popularizing a wrong idea of what real science actually looks like?
Re:A group of 3,000 citizens is making better forecasts than... by JoeAltmaier on Wednesday April 02, 2014 @09:28AM (#7515321)
Its not clear they continue to be 30% better - just that they started out that way, and got put on a team.

I'm still not convinced these folks aren't just outliers. People in all walks of life give far too much credence to lucky outliers - CEOs are hired that way, ball players, stock portfolios, sports predictors. If you make enough predictions and cherry-pick the best, you can always find a 'miracle man'.

Re:A group of 3,000 citizens is making better forecasts than... by jerf on Wednesday April 02, 2014 @09:29AM (#7515330)
"Its not clear they continue to be 30% better"

Yes, that was my point. The article failed to make it clear. It is worded such that it could go either way.

I agree they may be outliers. However, I can't dismiss out-of-hand the possibility that they are actually better. Outliers are common... but so are systematic/correlated weaknesses within large organizations.

Re:A group of 3,000 citizens is making better forecasts than... by mpyne on Wednesday April 02, 2014 @03:43PM (#7518394)
> but so are systematic/correlated weaknesses within large organizations

Indeed, that's such a big problem that there's a sort-of famous book written by a CIA analyst, "Psychology of Intelligence Analysis", that gives a pretty clear breakdown of how and why analysts fail to reason correctly about situations.

It also included examples of how Robert Gates (yes, the later SECDEF) managed to improve analyst reporting by actually reading the reports submitted and figuratively shredding the ones which were bad.

Before that there had been effectively no downside to submitting reports which were crap, intellectually speaking, and eventually things had become just professional-sounding educated guesses.

Large agencies don't have the benefit of small groups self-critiquing their own performance, so they have to try very hard to set and enforce standards of quality. When they don't do this it should not be surprising when quality drops.

Re:A group of 3,000 citizens is making better forecasts than... by danielrhodes on Wednesday April 02, 2014 @11:07AM (#7516021)
Total guess here, but you would imagine that since it is a project based on statistical prediction, they would continue to measure their ability to predict.
Re:A group of 3,000 citizens is making better forecasts than... by JoeAltmaier on Wednesday April 02, 2014 @12:38PM (#7516793)
Yes I look forward to a follow-up on this group
Re:A group of 3,000 citizens is making better forecasts than... by roc on Wednesday April 02, 2014 @03:33PM (#7518331)
> "According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information"

This makes me wonder how well analysts do, when they aren't given classified intelligence information. It's entirely possible that the analysts' skills are being hamstrung by information they think they know, but isn't actually true.

Re:A group of 3,000 citizens is making better forecasts than... by bane on Wednesday April 02, 2014 @07:57PM (#7519912)
> and I'd especially finger potential ideological biases there, which could affect both information filtering and the analysis

An excellent book on just this subject

https://www.cia.gov/library/center-for-the-study-of-intellig...