* * *

The wisdom of the crowds works given a large and diverse sample of independent predictors. People who don't know anything about a topic will vote randomly so their votes effectively cancel each other out, but people who know more about a particular topic will be biased towards correct answers.

"They're producing the wisdom of the crowd, which is a real and highly accurate piece of information."

I have strong doubts, that the wisdom of the crowd here is competent in judging whether a revolutionary new superconductor is real, or not.

what's an alternative that is better?

"They're producing the wisdom of the crowd"

Yeah we saw how wise was the crowd's wisdom with crypto.

This is correct - the overwhelming majority of people did not get involved with crypto. Even for people and companies who did most put a fraction of their money into it.

There is literally zero wisdom in the crowd about a brand new just discovered material that’s only ever been produced by one small group by definition.

This market fetishism is out of control.

That's not correct. Condensed matter physicists will have a good handle on how plausible this is (but not certain). Other people will vote randomly so their votes cancel out, effectively leaving the final result as biased by the expert opinions. That's how the wisdom of the crowd works.

So practically speaking, what can you do with the fact that X% of fans (because people betting are enthusiasts) think LK-99 will reproduce and Y% think it wont?

You assume all the people betting are enthusiasts. The theory of prediction markets is that rational actors in the market will recognise that a portion of the betters is overhyped and adjust their bets to make use of their irrational behaviour.

If the rational actors are actually effective at making such adjustments I don't know, I bet there's statistics out there on how well prediction markets correlate with reality.

In any case, even if the market was perfect, it wouldn't tell us if LK-99 would reproduce, which I guess is the meat of your question. It would just tell us how likely it is that an experimental result made under those specific circumstances would reproduce. And what you could do with that information depends on what your answer to the question: "How would I be affected if LK-99 would reproduce?" would be.

If you're a big energy business leader, and you want to filter what topics to spend your valuable time on maybe you could set a rule that you only want spend time reading scientific papers that have >10% odds of being legit.

More realistically though, I think things like prediction markets are mostly useful to traders who are trying to arbitrage things like resource markets. What's the price of copper going to do when this turns out to be true? You could adjust your futures based on that.

Have you heard the one about the Emperor of China's nose?

https://imaginatorium.org/stuff/nose.htm

Basically, making an average of a large number of estimates of an unknown value will (of course) fail if most/none of the estimators have any idea of the actual value being estimated.

I don't think I've seen that before. But the article you linked doesn't make the conclusion you suggest at all, instead they pose a corrected value. If indeed no one estimating had no information at all, the average length of Chinese person's nose would be close to that corrected value.

It's the same with this topic. You won't get an answer to the question "Is this particular paper true or not?" but you'll get an answer to the question "Are papers submitted under these circumstances making claims like this likely to be true?". The crowd will only answer the question they can answer. I think that's from "Thinking fast and slow".

It's not well explained in the version I linked (apologies, I should have looked for a clearer version).

The point of the story when I originally heard it is that no one has SEEN the Emperor's nose. So any statistical function (like averaging) of estimates is totally useless as they are all guesses.

No one has seen 'papers submitted under these circumstances' so no amount of 'crowd wisdom' will make any difference.

Also, as an aside the idea that 'the crowd will only answer the question they can answer' is ... bizarre. People will answer anything you ask them, and you have no way to know if they are just making it up.

> most/none of the estimators have any idea of the actual value being estimated.

A subtle distinction is who is allowed to participate in a prediction market.

"Everyone with $1" is a terrible answer, and produces the bad results people are pointing to.

Financial markets avoid this because of their scale, where there's enough smart money to (usually) punish stupid money.

Absent that scale, it's just stupid money muddling the decisions of smart money.

Prediction markets with a knowledge barrier to entry would produce better results.

How would you construct such a knowledge barrier? Another prediction market?

Also, suggesting that there is such a thing as 'smart' money - presumably due to having more of it? - is amusing. As pointed out elsewhere in this discussion, there has been a lot of smart money acting particularly dumb over the last few years.

A "Do you understand what superconductivity is?" or "Do you have a physics or engineering degree?" barrier?

And the relevant question isn't whether 'smart' money does dumb things: it's whether 'smart' money does dumb things less frequently than a random sample of money.

No one is an oracle, and there are absolutely outlier events that specifically confound experts, but I can't believe that increased expertise is uncorrelated with increased accuracy.

Tetlock’s Superforecasters performed better than experts, though: >In the Good Judgment Project, "the top forecasters... performed about 30 percent better than the average for intelligence community analysts who could read intercepts and other secret data"

> They're producing the wisdom of the crowd, which is a real and highly accurate piece of information.

Unless the group of people is not a crowd but rather a mob.

The opinion of a crowd is generally useless in highly technical matters. The people betting on this stuff generally do not have the background to evaluate any claims appropriately, and just react to what other people (who they believe to be more informed) are saying.