News for hackers. Stuff that matters.
Re:Randomization in such studies is arguably a negative in p... by tgv on Friday April 05, 2024 @02:50PM (#39945730)
The money quote at the end: Randomization doesn’t make a study worse. What it can do is give researchers and consumers of researchers an inappropriately warm and cozy feeling, leading them to not look at serious problems of interpretation of the results of the study, for example, extracting large and unreproducible results from small noisy samples and then using inappropriately applied statistical models to label such findings as “statistically significant.”

Spot on, unfortunately.

Re:Randomization in such studies is arguably a negative in p... by waterhouse on Friday April 05, 2024 @03:25PM (#39946113)
I'll note that one could say the same about any specific feature that improves a study. There's a list of ways that a study could go wrong, and if nine aspects are good, that doesn't rule out the tenth being awful enough to torpedo the conclusion.
Re:Randomization in such studies is arguably a negative in p... by Carlap08 on Friday April 05, 2024 @04:08PM (#39946614)
Unfortunately nowadays people seem to have the same understanding of the terms "statistically significant" and "proven causality". There's no way a single study can prove causality and the closest we can get to reaching for the truth is meta-analysis.

Not saying studies results are invaluable. They can certainly point to the right direction if done right. But just a single RCT won't be enough to prove causality.

Re:Randomization in such studies is arguably a negative in p... by svara on Friday April 05, 2024 @04:18PM (#39946703)
> There's no way a single study can prove causality

Why? If you're talking about epidemiological studies looking for associations between variables in big datasets, okay. But that's just one type of study, and it doesn't get better by throwing many of them together in a meta study.

You absolutely can show causality in a single study by doing experiments.

I'm feeling rather the opposite really, "correlation is not causation" is a beaten horse and will come up in every discussion where it may or may not fit on this website.

In many cases, correlation is a hint at causality, and if there's a strong model explaining it, should be considered evidence for causality.

Re:Randomization in such studies is arguably a negative in p... by bsder on Friday April 05, 2024 @04:47PM (#39947023)
> if there's a strong model explaining it

This is the key that everybody ignores.

If you simply throw variables together, you wind up with too many confounders. You can also get the direction wrong.

"Growing tall causes you to be male" is my go to example. The correlation is super powerful--because it's completely backwards. It is also competely valid unless you have a model to differentiate--at which point it becomes obviously non-sensical.

Re:Randomization in such studies is arguably a negative in p... by mquander on Friday April 05, 2024 @07:13PM (#39948394)
If you ran an RCT where you could directly make people to be male and then see what happened to them, you wouldn't need any further modelling to distinguish between which is causing which. The reason you need the model is if you can't randomize.