They used a prediction market, which at the time the article was published was still to resolve. This is standard practice and methodologically sound. They didn't go around asking people on Twitter.

Just skimmed it quickly, but from [their paper](https://royalsocietypublishing.org/doi/10.1098/rsos.200566 ):

> Participants were recruited through a number of mailing lists, Twitter and blog posts. [...] For all analyses, we use survey responses; for analysis (iii), we additionally use responses to a demographic survey that included a question on academic interests; and for analysis (iv), we additionally use the prediction market data.

So it sounds like they only used prediction-market data "for analysis (iv)", which they describe as:

> [...] (iv) whether survey-based aggregated forecasts and market-based forecasts are correlated.

---

The last line of their paper:

> If our forecasts hold up, it will be interesting to investigate if specific factors (such as different methodologies and policies) can be identified that influence replication rates.

So, it doesn't sound like they're claiming that their forecasts should be assumed reliable.

No, it isn't. A prediction market is not proof of anything.

I didn't say that the methodology was not standard practice or methodologically sound. I said you misrepresented the study, which you did. You said "Most social sciences have numbers that are far better than psychology's." in the context of a conversation about *replication numbers*. You did not say "in a prediction market, participants ("recruited through a number of mailing lists, Twitter and blog posts") have lower replication expectation rates for Psychology than other social sciences" -- if you had then, well, you wouldn't have misrepresented the study.

Any meta-study trying to get replication numbers would likely use a combination of methods, including prediction markets, with techniques like having a small fraction of the contracts resolve.

So no, implying this is a Twitter poll is a far worse misrepresentation than even the most uncharitable interpretation of what I said.

> Any meta-study trying to get replication numbers would likely use a combination of methods, including prediction markets, with techniques like having a small fraction of the contracts resolve."

this is non-nonsensical obfuscation.

poor attempt at moving the goal posts -- for the record I did not imply, I DIRECTLY QUOTED from the article.