the reason it's the gold standard for medicine is because the only thing that is different in aggregate between the groups is the intervention. Therefore, it is the strongest possible evidence of causality we have, and does say something about causality (rather than correlation).
It's important to replicate, but the points made by Pearl are not relevant here and the mechanism is moot when you have a causal link shown by RCT.
http://andrewgelman.com/2018/01/08/benefits-limitations-rand...
To be honest, I think a randomised clinical trial is more of a starting point than an endpoint in terms of 'knowing' that an effect is genuine. The 'gold standard' moniker overdoes the authority of the RCT a
Correlation != causation. Which is why you need to do randomized testing and try and account for other causative factors. But if you give a bunch of people a medication and they stop dying, you can pretty much say this is valuable and start giving it to people.
Others may value the mechanism more for very legitimate reasons. For instance, if you underatand the mechanism it leads to further research that could utilize a related mechanism or perhaps see potential side effects that would otherwise be hard to identify.
You really need both. "Is it true?" is a great starting place and oofers some treatments, but discovering the mechanism gives potentially a whole new class of treatments.
the problem is that efficacy studies (like I want) are not that exciting to report. as a result, the lay press reports only mechanism studies (because their readers like to be "wowed" and think about things, like the poster I originally replied to).
that wouldn't matter at all, but we all get sick. and when a person gets sick with this mentality that they need to understand the mechanism, then they start overthinking the treatments that are available today, drawing all kinds of spurious conclusions, and making medical decisions that are unjustified at best and harmful at worst. and in my experience, the smarter the patient is, the more willing (and insistent) they are to take this approach (witness Steve Jobs). This is absolutely not the patient being "dumb" or stubborn: it's completely understandable given how they understand science. But it hurts them, which no physician likes to see.
let's note how incredible it is that we are having this conversation in the setting of some incredibly expert scientists (albeit in other fields). it really shows how hard science can be to understand sometimes.
There is a huge amount of effort required to say why something works with certainty. Usually to get there requires many years of experience with a new surprising result. Which requires publishing lots of results before understanding the mechanism.
If you refuse to believe anything that isn't figured out from first principles is true... I mean... I have a condition and if someone told me a higher percentage of patients on a new drug did well than on the one I use, do I care if they can tell me exactly why? Not in the slightest. I will wait a year or two to see if side-effects crop up and ideally see the results confirmed, but I'm not going to not get a treatment likely to help just because they don't know how exactly how it works.
I think it's good practice to assume the results of a paper are accurate, even if you give no credence to the interpretation.
The motto of the Royal Academy of Sciences is quite literally "take no one's word for it". If you wish to blindly swallow assertions made by people with academic titles, perhaps you should start to listen to those who advise against it.
Says who? ;)
> Your heuristic may be useful, but it is just that - a heuristic, not an argument.
Indeed, but in many but the most trivial cases (stuff like newtonian physics, or computer science, where most arguments are easily falsifiable) heuristics is all we have. Just look at the set of comments we are currently part of, e.g. many of the p-value posts, or the criticism of the small sample size. Those are valid arguments, but also wrong. But how can someone not trained in empirical science find out? Relying on an authority (relevant to the topic at hand) is a valid way to be wrong less often.
To put it pointedly: How do you know its always fallacious? Have you actually proved it yourself, or is this from a textbook/the internet, e.g. - from an authority? ;)
http://slatestarcodex.com/2013/12/17/statistical-literacy-am...
Ask your local doctor next time you visit to tell you what a P value means.
The medical literature is a tragicomedy of statistical incompetence and malpractice.
Specifically on this study it has a lot of red flags: not pre-registered so p hacking can be assumed, industry sponsored which studies show greatly skews results, small numbers, unrealistically high effect sizes.
http://andrewgelman.com/2017/01/30/no-guru-no-method-no-teac...
I think the way to combat this is to avoid studies that solely attempt to demonstrate an effect, but to complement that part of an investigation with additional studies that demonstrate a plausible explanation. This reduces the risk of false positives and increases our confidence in the result.
The additional studies could take the form (for example) of deliberately intervening such that the effect should be manipulated by the intervention in a predictable and measurable way. Essentially, come at the hypothesis “from another angle”.
there are tons of potential pitfalls and a combination of
experimental limitations
you provided some, which are at least partially addressed by the study design. (I don't think it's your point, but let me agree that "the scientist can cheat" is always true). poor academic incentives,
this is addressed because it is double-blind insufficient statistical and scientific training
this is (partially) addressed by peer review, and the availability of the data. you can't make this general criticism of the paper at hand: find the place where the design or statistics used were in error. I think it’s bad science, that will one day
retrospectively be viewed as invalid scientific method.
that's a claim without any evidence to back it p. I think the way to combat this is to avoid studies
that solely attempt to demonstrate an effect, but to
complement that part of an investigation with additional
studies that demonstrate a plausible explanation. This
reduces the risk of false positives and increases our
confidence in the result.
If you have a double-blind, placebo-controlled, randomized clinical trial then what can the mechanism possibly add? Is an effective medication less effective because we don't understand how it works? If you want an amazing example, read about the mechanism of action of general anesthesia: we have many ideas about how it works, but nobody really knows. Nobody gets caught up by this when their appendix ruptures. I hope, for your sake, that you will lower your standards in the event that you need surgery. The additional studies could take the form (for example)
of deliberately intervening such that the effect should
be manipulated by the intervention in a predictable and
measurable way. Essentially, come at the hypothesis “from
another angle”.
If you have a properly designed and executed randomized, double blind, placebo controlled trial, then you have measured the effect of the intervention, and only the intervention. You can repeat it in order to gain confidence, but you have to decide when the diminishing returns of "more confidence" are not worth your study dollar. And for curcurmin, which is cheap and has low side effects... it's probably not that far away.edit: fixed formatting
> that's a claim without any evidence to back it [u]p.
You are correct. The irony is rich here.
I always welcome a good argument (whether in dispute or affirmation of an issue), but just saying "I think <XYZ>" doesn't make it so. Reading through this thread, all I see is the common HN Default Skepticism / Negativity -- but more so; skepticism with less scientific rigor than the work being criticized.
A mechanism is really just a narrower example without irrelevant distractions or confounding variables.
Kind of like the difference between the bug reports: "run this complex workload on this specific machine and it will orobably crash after a few hours" versus "run these three commands when the cache is cold and it crashes".
What is your view on science, do you think it can create - and verify - a true result? Whither axiomatic underpinning in such a view?
I think it's clear that science gets us truth, but that the mechanism for that complex social activity can't be reduced to a few axioms or principles; Feyerabend gives a detailed description [1] of how bad a scientist Galileo was (as judged by modern theories of what science is), yet he was right, and he advanced truth.
[1] Paul Feyerabend, Against Method (1975)
Heliocentrism isn't truth, it's opinion [that's my opinion, lol!!]. Einstein's relativity makes heliocentrism false in demanding that there is no aether and so no special points in space.
Science provides our understanding of a best current description of reality - the whole point of scientific endeavour is to demonstrate we were wrong and that our understanding/description was flawed.
Moreover it relies on maths, which is provably unable to verify it's own consistency and completeness.
Furthermore it's an axiomatic system that produces a probabilistic framework that gives us little to hang our conception of reality on [what I'm saying is our perception is not particularly informed by physical reality, we have to subdue our instinctive perceptions to work with what amounts to our best, most consistent, physics view of reality].
Science gives us as much truth as we can see in a well lit test-tube.
significantly slowing cancer spread with for example 4 months survival result - less than 40% of controls and more than 80% of baking soda drinker mice https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2834485/figure/...
The study has all the statistical bells and whistles. And its result perfectly matches our understanding of cancer spread process - local overproduction of acid from glucose aerobic glycolysis (dominating way of glucose consumption by cancer cells) results in dissolution of the ECM (which is mostly collagen) - and the probable mechanism of action of the baking soda on it.
More generally, however, a plausible mechanism is a great aid in determining the potential value of further researching a correlation because as we all know, correlation != causation, but correlation + mechanism pretty much is causation, which means correlation + plausible mechanism is very interesting. It's not wrong to ask "how?" here, though it might not yield much given the nature of the field. Certainly, anti-inflammatory properties can plausibly have positive effects on the psyche, so that's a mechanism worth some looking into.
For example, How meaningful is the 20.3 point difference in Buschke SRT test? Is it noticeable in daily life, or just something detected in testing.
In this trial, and 99% of other trials, there is no mechanism provided by the study, only a measurement of causality (which is the same thing as effectiveness).
The authors may think they know based on other, prior, basic science research (as they are postulating here), but it does not affect the conclusions of the trial ( whether they do or don't (or even if they are wrong).
Obviously it's intellectually satisfying to understand WHY, but it's not as important as "is it true?"