This quiz will let you test how accurate you are at predicting which psych papers replicated or not. Last time I took it, I think I was at 8 or 9 out of 10. I think something close to 50% of such papers don't replicate - so it's probably a reasonable prior to just assume any given social psychology study won't replicate.

https://80000hours.org/psychology-replication-quiz/

> so it's probably a reasonable prior to just assume any given social psychology study won't replicate

I'd pay that. Though in this case it's not people reading the paper, but reading the article about the paper. (On reading the paper, the findings are less conclusive than the click-baity headline, but that's almost always the case)

It's not about an Oracle. It's that the mentioning of a topic prior to some conclusion generating observation has practically never generated replicable results. Still, these studies make news all the time because they're easy to p-hack into some interesting conclusion.

Reading the study it appears they went to some pains in not revealing their intentions:

> The caregiver and child were seated across from one another at a table. The child completed an unrelated experiment with the researcher, while the caregiver completed the Scarcity or Control manipulation survey on a tablet. When the caregiver had finished, the researcher left the room under the guise of loading a second survey onto the iPad, leaving the caregiver and child alone with a toy the experimenter happened to offer as she left. A video camera and/or tape recorder recorded their interactions.

The conclusion could be that being reminded or queried about hardships has a chilling effect

People are good at predicting this: https://www.theatlantic.com/science/archive/2018/08/scientis...

> People

To be specific:

> They recruited 206 volunteers—a mix of psychologists and economists, students and professors, none of whom were involved in the SSRP itself

Also to be specific, none of the volunteers needed to consult an Oracle. The factors they cite (low N, "newsworthiness") are also the same factors that would give a layperson pause.

I think if we're invoking the reproducabilty crisis, it's a bit ironic to then dismiss demographics

No, not really.

The point is that it's possible to relibly predict whether a study will reproduce based solely on the characteristics of the study.

And a high portion of the cohort of that study were people that read academic papers as an occupation. I'm not saying that the study couldn't generalize to lay people, just that you haven't presented evidence of it

(I believe one of the early controversies that sparked the reproducability crisis was the discovery that an excess of studies were using college students as their participants, skewing their results)