how do you prevent cherry picking?

that's such a vague question

for example, if you have 50 partecipants but only provide the multimedia evidence for 20 of them your study should be thrown out the window

What is preventing someone from having 200 participants, but saying they only had 100 participants, and then only providing evidence for 100 participants?

The researchers will have to have gone through some kind of third party agency to get the partecipants. This agency should be queried to see the number they report

How does this agency determine who can meaningfully participate in the study? Are they going to have the expertise to make that determination for _every_ study that could possibly be conducted?

What is the difference (to a layperson) between cherry-picking participants and rejecting participants because they do not meet your study's criteria?

Who funds this agency?

Do the members of the reviewing agency have their own biases, and might those biases tarnish the reputation of a study that is actually well-conducted? (hint: this already happens in journals)

You have to declare the study population before you request the next round of funding. Thereby fixing the problem.

Nothing. Pfizer did this, albeit wasn't 50% of the enrollment.

You can see that here: https://kirschsubstack.com/p/pfizer-phase-3-clinical-trial-f...

No Pfizer did not do that.

Pfizer had a trial with 21823 people in the Expirement group and 21823 people in the Placebo group. In the results they excluded data for 1790 from the Expirement group and 1585 Placebo group. However, _crucially_ Pfizer never claimed there were only 100 people in the study after starting with 200; you know pfizer excluded 3375 people because _Pfizer told you_.

"But we have 20, the other 30 volunteers were removed by unrelated reasons" (and it is common to exclude volunteers from experiments)

Every clinical trial paper I've read has a discussion of inclusion and exclusion criteria. I think for the trial to be registered, it has to include this information.

"unrelated reasons" should not be an acceptable excuse though. either state the reason or it goes into the trash. and if they were private reasons you can still contact them to confirm they left on their own volition and/or they didn't finish the trial without getting into specifics. you only need one lie to suspect the whole thing

These are medical trials. How do you preserve the patients privacy in all of this?

Or do subjects need to wave all their doctor-patient privacy rights before joining any trial?

If we discover that we can´t trust researchers then what else are we left with? Doctor-patient privacy works if the doctor is truthful in their reporting

It's generally permitted to share de-identified patient data. As long as you're not sharing patients names, medical record numbers, birthdays, and a couple of other fields, you should be fine.

Maybe we could do a double blind (including scientists) study where everyone waived their rights & are recorded then in another "typical" conditions study do none of that and compare the two and see which one seems to have the best, most accurate results.

> maybe as an intermediate step we could make available all the recordings to the peer reviewers

The issue is clearly not the amount of data available to peer reviewers considering it's already easy to detect major flaws in a quarter of published peer reviewed research. The issue is that peer reviewers do a shoddy job which should surprise no one having ever published peer reviewed research.

And to be fair why should they do better? It's generally unpaid, it's poorly paid when it is paid and it's not particularly well considered.

Sounds like a YC idea?

LOL looks like someone has not had to get data collection protocols through IRB approval...

if researchers are so untrustworthy then what's your solution?

For a 'quick' overview into the mess that is IRBs, this book review is a good starting place: https://astralcodexten.substack.com/p/book-review-from-overs...

TL;DR: IRBs are a mess, hyper-individualized, and the problem ain't getting any better any time soon.

>it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments

This is a great approach IMO. Additionally skeptics (for example anti-vaxxers), should be physically present at the trials.