It's important to keep in context where the term "Wisdom of the Crowd" came from and where it succeeded.

"The classic wisdom-of-the-crowds finding involves point estimation of a continuous quantity. This has contributed to the insight in cognitive science that a crowd's individual judgments can be modeled as a probability distribution of responses with the median centered near the true value of the quantity to be estimated." [1]

Notice that the term said nothing about adversarial games and or games which have moves.

A better model for those types of games is the experts' problem / multiplicative weights algorithm -- where the solution consists of a weighted plurality which is updated every turn. [2]

[1] https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

[2] https://en.wikipedia.org/wiki/Multiplicative_weight_update_m...

I cannot agree.

This sort doesn't prove nothing about "Wisdom of the Crowds" because the wording is too broad. But it is evidence that crowd can outperform the best individual at least theoretically. The worst mistake the crowd had made was openness of dicussion allowing Kasparov to read this discussion. It gave him the advantage and by his own words it was cruicial advantage to win.

Chess is the good candidate for hive-mind, because it is computational task which can be parallelized. You can hire tons of minds, and made them to solve different parts of the common task. There was some issues with management with mind-resources, the worst at the end-game, when many people lost their faith and started to discuss not a next move, but what is wrong with hive-mind.

> The most rigorous experiment I've seen performed, tested a group of reddit users on estimating lines in an image [3]. Some expert individuals were far more accurate than the group average.

I'm not convinced by this. There are few objections.

1. There was no discussion between elements of hive-mind. So the abilities of hive-mind remains unused.

2. Suspicious statistics. a. The articles compares estimates of individuals with estimates of hive-mind. Hive-mind have one set of answers, while answers of individuals are combined with knowledge of true answer. There are no example of individual outperforming hive-mind on all four data points at once. b. There are no attempt to estimate probability to get top individual answers by chance. Maybe those selected individuals are no better than average person, but just lucky ones?

3. There are different tasks. Different tasks needs different data processing, different data processing needs different processors. For example, there are tasks that benefits from parallelizing, and there are tasks that do not. Here we come close to (1): if you want hive-mind to outperform individual, you need to find a way how to use superpowers of hive-mind with this task.

Wisdom of the crowds works when you're able to pool a large number of independent, unbiased estimates to produce a distribution. If the crowd communicates, it is no longer independent. This produces a distribution weighted towards the answers of loudest or most persuasive fraction, which is probably going to be less accurate.

In fact, communication between crowd members in many cases significantly reduces the quality of the answers. This has been clearly demonstrated in studies of group brainstorming, where the number and quality of ideas is significantly better when group members come up with ideas independently and pool them at the end than when they come up with ideas together.

Perhaps Deep Blue (or current equivalent) vs the World would be an interesting match, where the AI did not have access to the discussion.

Proves?! It isn't even an anecdote in favor of your thesis. The World Team stayed with the world champion until move 54 (something proved 13 years later).

Avoid over-generalizing.

To this point, consider the failed startup Trada. (I worked there for a bit)

They had a marketplace for ad campaigns with Google. Instead of running ad campaigns or using artificial intelligence, a pool of optimizers would work on a campaign and Trada worked as the middle man between optimizers and advertisers.

The Trada logo was a jar of jelly beans: (http://wisdomofcrowds.blogspot.nl/2009/12/jelly-bean-experim...)

> A classic demonstration of group intelligence is the jelly-beans-in-the-jar experiment, in which invariably the group’s estimate is superior to the vast majority of the individual guesses. When finance professor Jack Treynor ran the experiment in his class with a jar that held 850 beans, the group estimate was 871. Only one of the fifty-six people in the class made a better guess.

When advertising on Google competition leads to higher prices. For example the keywords "dui lawyer" cost a fortune because the rate of return is so high for each click. Theoretically a creative individual could come up with a set of untapped, low price keywords that could lead to an effective ad campaign at a lower price.

In practice the company had major problems. Incompetent optimizers would chew through a whole budget, leading to advertiser churn, and a lot of intervention. Eventually everything becomes micro-managed which isn't a sustainable business model. (There's a reason agencies target big ad campaigns... its the only way to make it worth it because of the amount of resources you have to invest)

So there's a single data point where the wisdom of the crowds failed.

I'm not sure I understand how incompetent optimizers leads to an example of the wisdom of the crowds failing. It sounds like the failing you describe is the result of individuals. Is there something I'm not getting?

The optimizers were the crowd in this business model. Some of them made good decisions, others made bad. But since the amount of money available was limited, those bad decisions brought down the performance of the campaign and ultimately the advertiser gave up.

An expert would've made better decisions.

I think the wisdom of the crowd can sometimes be effective, but there are a lot of ways it can go wrong. Some things I was thinking of:

1. the crowd wasn't big enough

2. the time frame was too short to be effective

3. the heuristic was too complex or poorly tuned to lead to a good outcome

4. creative endeavors with high rates of failure lack enough signal to properly optimize (a strategy might look terrible till it suddenly works)

5. misaligned incentives undermine the goal

To illustrate the last case, in high school I had a teacher tell us that he would scale the test by the highest score. For example a 95 would give 100 and +5 to every other score in the class. With incentives like that, if everyone in the class answered no questions, everyone would get 100. (but then all it would take would be one individual to screw the whole class)

Crowds are complex, difficult to understand and hard to predict.

On "Wisdom of the Crowds": Group average on its own is no good, but some creative aggregation can be better than either member on its own (and of course better than naive averaging of any sorts). As a practical example, see for instance Metaculus. Excerpt from one comment[1]:

"Since Dec. 1 2018 (sic), the MP has been on the "right" side of 50% every single time, and (in a more meaningful measure) has a mean Brier of 0.048. (Though those are both about to be spoiled by the NASA-LISA question, probably.)" [2]

You can compare metaculus vs. community predictions at [2]. The corresponding community score would be 0.119 during the same time.

[1]: https://www.metaculus.com/questions/666/will-this-question-r... [2]: https://www.metaculus.com/questions/track-record/

I'm not sure this is proving anything except that by developing an algorithm that filters out some predictions you can improve upon those predictions. And that is probably quite obvious given there are some users (presumably) on the site who are just trolling and fairly easy to identify and filter out. Unfortunately, I can't see the data on the Brier score of individuals.

Of course it does not prove anything, but identifying "wisdom of the crowds" only and exactly with "group average" isn't particularly considerate. While public mass media thrives on simplifications, I had hoped that we could iterate on the basic premise.

Quite the opposite, actually. The world team played better than anybody from that team participating individually could have.

Yet wisdom of the crowd becomes a successful machine learning paradigm as ensemble learning where a family of weak learners are combined to produce a stronger learner.

Read the comment by "Someone". The rest of the world was playing on both sides of the table, since Kasporov was reading their analysis during the game.

Or maybe it means you have to curate your crowd rather carefully if you're into outperforming arguably the greatest chess player ever.

"Wisdom of the Crowds" works really well for some problems, and not at all for others.

Wisdom of the crowds works well for counting jelly beans in a jar, not so much for finding answers to much more complicated questions.

Relevant quote: "In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual."

Checkmate, democracy.

I tend to think democracy is less about finding some optimal solution and more about finding a solution enough people will be happy with that there isn’t revolution.

Yeah, democracy may have a limited best case scenario compared to other forms of decision making, but in many senses it has one of the best worst case scenarios. More centralized power may be able to get more done, but historically we've seen it often results in decision making that is either just-plain-bad (when leadership is clueless) or which only benefits those in power, to the detriment of everyone else (when leadership is malevolent). Whereas democracy is slower and less efficient, but does ensure that most people are in support of the decisions being made

Democracy isn't about getting the best policy each and every time, its primary value lies in the fact that it creates framework that facilitates the peaceful transition of power and avoids the otherwise frequent wars of succession, purges, coups and whatnot.

democracy is not about decision, but decision making. during the process consensus is born and grown. it then generates mandate and enough political capital and willingness for consequent execution.

that being said, democracy often leads to wishes that nobody could execute. after all, it's about what participants want, not what they really need.

Democracy does not try to make good decisions, it tries to reflected the opinions of the people in the decision.

I don't think democracy is about increasing intelligence so much as resilience and escaping from failure modes. It provides a way to get rid of bad leadership, and it makes certain that huge issues are hard to ignore. In a pure autocracy a bad autocrat is fatal and there is no way to do anything about it but via extralegal means.

Trump was the checkmate

The two most rigorous experiments in the wisdom of crowds are:

- the stock market

- democracy

They beat technocratic efforts every time.