From the abstract: "After additional adjustment for race/ethnicity, income, education, self- and physician-rated health status, body mass index, leisure exercise, smoking, and regular alcohol use"
> On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?
How do you suggest studying causation in this setting? A randomized controlled trial where we deprive people of health insurance? Even that will likely not yield a true causal estimate, because randomization only helps for pre-randomization differences in the population, and behavior change from lacking health insurance will occur post randomization. The authors do extensively discuss their control variables, and important to remember is the fact that most papers only control for variables which ended up doing something, a subset of all variables that were tried. The NHANES data the study was pulled from includes a staggering number of covariates.
---
While not an ironclad study, I found the paper itself vastly more compelling than the politifact analysis of it, which boils down to "Well, observational studies might be wrong because reasons".
Yes.
Before instituting Obamacare/Romneycare/$POLICY, we should have run a pilot program based on random assignment with clear predefined success metrics. But that's politically dangerous - after all, what if the experiment shows that $POLICY doesn't work?
We did that, by accident, in Oregon (google Oregon Health Experiment). There were no statistically significant results beyond the placebo effect [1]. Strangely, none of our fact based politicians have proposed scrapping the medicaid expansion based on that.
[1] People with insurance perceived themselves to be healthier before actually consuming any medical care and became less depressed. But no statistically significant difference was observed in any of the objective metrics chosen before the study started.
As for the Oregon study, the results of that study are still relatively new (the idea that any measure focused on preventative health will show results after two years is pretty suspect). The authors of the study discuss this for diabetes:
"Medicaid significantly increased the probability of being diagnosed with diabetes after the lottery (by 3.8 percentage points, relative to a base rate of 1.1) and use of diabetes medication (by 5.4 percentage points, relative to a base rate of 6.4). As discussed in the paper, based on clinical trial evidence on diabetes medication, we would expect this increase in the use of medication for diabetes to decrease the average glycated hemoglobin level in the study population by 0.05 percentage points, which is well within our 95% confidence interval for the impact of Medicaid on the level of glycated hemoglobin."
As for the number you are cherrypicking, it is true that health insurance increased medical consumption (including ER visits, in spite of what ACA supports claimed) among people who received it. However, no measurable effect on health (besides depression) was observed.
The original study that showed aspirin's effect on heart attack prevention comes to mind for the former circumstance, and a number of HIV prevention studies for the latter.
Either way, no point in continuing this.
Beyond that, in an intentional randomized trial, rather than the 'happy accident' like the Oregon study, the actual control is not 'Nothing' but the medically indicated standard of care. Studies are often required to provide medical care, education, etc. to their participants. I cannot imagine a study managing to get "We deny a bunch of folks health insurance" by an IRB unless it was an externally forced process, like the Oregon study.
Your insult about not understanding statistics, in addition to being off-base, is rather spurious. This isn't a statistical question, it's a public health ethics question. Statistics doesn't really come into whether or not "Keep a bunch of people from accessing healthcare" will get nailed by an approval board.
Also, the FDA often does take observational evidence into account, especially when expanding things like what age range a drug is medically indicated for.
"Your new homepage goes viral, but you aren't sure what copy is converting. Hook that copy up to BayesianWitch and only converting copy will be showing. No waiting days for the answers from an A/B test."
Seems to imply a reliance on "correlational" data. Is there some hidden randomization in there? Or are you giving your clients a lower standard of evidence?
Similarly, if you have 10,000 seo-optimized microsites, each with traffic too low for a per-site A/B test, we'll improve your conversion rate across the 10,000 microsites.
If you want to make a long term change (e.g., logo, button color, feature) for a high traffic site (your one and only landing page) you are better off using a traditional A/B test.
A lot of uninsured people can get medical care in emergency rooms which are not allowed to turn them away, but then the cost adds one more huge burden on top of an already-difficult struggle to get out of poverty.
Which might be the case. You argue that Oregon showed that. Doesn't that paint medicine as a huge fraud, regardless who is paying for it?
So that's part of what happened - if you look at the data, both the control and treatment group did consume medicine. You don't need insurance to get treated. But medical consumption increased in the treatment group - it just didn't improve health. That suggests medicine has a point of diminishing returns, and people without insurance already consume enough to reach that point.
(Also, a caveat: the Oregon Experiment was too short to measure an effect on life expectancy. They measured several other proxy health measures instead.)
Someone who works for 'Bayesianwitch' should know better than to rely on p = 0.05 as the sole basis on which to evaluate something.
I would have said that looking at chronic health outcomes after just a few years was probably a losing proposition, and asked to see some power calculations, or a longer term analysis plan.
I've done so for other studies. Was actually grousing about one in a meeting...two weeks ago?
I'm also skeptical of Politifact's competence to adjudicate public health scholarship.
I'm actually not sure the study authors should be blamed.
But: given how politicized this question is, they could have reasonably anticipated the misuse of their results, and thus could have written their results in such a way as to avoid this. Or they could have publicly corrected non-experts who cited them for causation. Or: they could have controlled for factors that would make mortality and insurance-status independent. This last option is difficult & requires complex judgement calls (see [1] for a reasonable attempt), but even if you feel the other two aren't required of academics, this last one very much may be.
(Separately, I agree that Politifact should not be trusted automatically, but these seem like reasonable analyses, and I did not quickly find anything better.)
This is an appeal to authority. And unfortunately, the evidence is accumulating that there are problems with that authority. The success of peer-review depends on the quality of the peers and of the review. If those reviewing don't understand how to evaluate a correlation study, or do understand but don't take the time to properly evaluate it, then garbage will slip through.
It turns out, lots of garbage is produced and sent to the reviewers, as was noted in a recent Nature feature:
http://www.nature.com/news/scientific-method-statistical-err...
see also:
http://www.nature.com/news/weak-statistical-standards-implic...
http://www.nature.com/nature/focus/reproducibility/index.htm...
---------
tl,dr; lots of crappy correlation studies are published in peer-reviewed journals. These studies later turn out to be irreproducible.
Also, while "Correlation is not causation" is, as you mentioned, a tired canard on the internet, people often seem to forget that all things that do have a causal relationship with have some form of association. Association doesn't prove causation, but it's a damned fine first step, and miles above "guessing", which is what happens without evidence.
Probably taking away insurance from people who have it as an experiment is too extreme, and not implementable anyway. But is it really that unethical to take a subset of population without insurance, give it to a random subset for some time, and observe the differences? Why? No one from the control group is prevented from getting insurance on their own (compare with the candidate drug trials, where the control group can't just buy the drug on their own).
No, it's not set in stone, but if you're talking about implementing studies now, you're not going to get major medical ethics reform first. You go with the system you have, and the system you have is probably going to push back pretty hard.
> No one from the control group is prevented from getting insurance on their own
This alone is a difference between the control and treatment groups that takes place post randomization, and knocks said experiment back into the realm of "correlational"
So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.
There are all kinds of things that we cannot prove, because it is either impossible or wildly unethical to conduct a randomized study. For those things, you can make a determined effort to control for as many "third factors" - the technical term for them is confounders - and that gives you a level of evidence which is well above guessing.
Since I can't reply to your comment, my responses here:
> "You didn't say proof but you said it's better than guessing, and I don't agree with you at all."
It is better than guessing. You're welcome to disagree, but a well conducted observational study is considerably firmer evidence than pulling it from your posterior.
> "What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?"
All you've done is describe a really bad study. You can have really bad RCTs as well, by the way.
Of course you question whether or not an observed association has a clear biological or social mechanism. And you attempt to control for other variables that might influence the link between your exposure and your outcome. You run followup studies in different populations to try to understand if the result is a widespread phenomena, or a fleeting bit of statistical noise.
Basically, you do your job well. Which is why I used phrases like "a good first step".
Your example is about as useful as "Programming is useless because once I coded something poorly and corrupted my data".
Correlation is useless, and there's a ton of observational studies out there finding correlations every single day for which we have no rational explanation at all. Observational studies are full of variations in the way they are designed, the way they are reported and the subjects of the studies, it's rather a miracle if you actually detect a hint of causation based on the garbage noise that you get.
How can you tell if you missed one?
> [...] gives you a level of evidence which is well above guessing
> It is better than guessing
Why? Without any support, this seems to be an appeal to probability.
> a well conducted observational study
> a really bad study
The fallacy of moving the goalposts; also the no true Scotsman fallacy.
Similar studies, using as many variables as you can find. Residual confounding is always and ever a problem, but the odds that something is both a strong residual confounder and has never been observed to have an association with the outcome or the exposure is pretty rare?
> Why? Without any support, this seems to be an appeal to probability.
It's really not - if for no other reason than it's forced you to think about your system more than a simple guess would. It's not an appeal to probability, its using data to update whatever prior you came in with. Guesswork is just using your prior.
> The fallacy of moving the goalposts; also the no true Scotsman fallacy.
Not really, no. Some observational studies are crap - this is just true. But that doesn't say anything about the potential quality of observational evidence, and many of the commonly raised objections to observational studies are actually objections to poorly run studies. The example used was a study that examined no potential confounding variables, looked at a correlation with no prior evidence suggesting any linkage between the two or biological plausibility, and then asserts that they've found a causal link.
That's a bad study. It's not 'No True Scotsman Fallacy' to say that the problems with a bad study don't generalize to all studies. If it is, then we're all screwed, because you can run a bad RCT too.
Appeal to probability; appeal to common sense.
> Some observational studies are crap - this is just true.
Special pleading.
> But that doesn't say anything about the potential quality of observational evidence
Straw man.
> many of the commonly raised objections to observational studies are actually objections to poorly run studies
> That's a bad study.
Moving the goalposts. No true Scotsman.
> It's not 'No True Scotsman Fallacy' to say that the problems with a bad study don't generalize to all studies.
Straw man.
This is a totally unreasonable stance to take. You can't even imply anything at all with correlation? Really, nothing at all? It's no better than a random guess? Try actually doing some actual science with this attitude, and keep to it consistently, and let me know how far you make it. In fact, try the same thing with ordinary life, any kind of reality where your decisions have consequences in reality.
FTFY
Yeah, thanks for fighting for the cause! #sarcasm
Is HN supposed to be a place where we try to find a flaw on every statement and make sure it doesn't go unnoticed?
What you pointed out doesn't even diminish an ounce of what Brandon and his teammates are doing.
A recurring and important (to me, anyway) question here is how technical skill may be leveraged to provide real value to the world. This is a hard and unanswered problem. This [brandonb's] statement is so strong that, if it were true, it would eliminate a vast territory of alternate [possibly correct] paths to answers ("oh, you did X? My code saved 50 lives this year."). So it [brandonb's statement] is worth challenging (or correcting) more so than any random statement here. And yes, I do view promoting accuracy (even disillusion) as fighting for the cause.
I hope Brandon will accept my sincere thanks for working on something that is important. Irrespective of health impact, the financial risk borne by the uninsured is an important issue and not controversial.
edit: clarifications in []
"This is a hard and unanswered problem. This statement is so strong that, if it were true, it would eliminate a vast territory of alternate paths to answers ("oh, you did X? My code saved 50 lives this year.")."
It seems like you are more of the problem rather than Brandon's statement. No matter what Brandon says, accurate or not, people will still find a flaw in it. People will see what they want to see.
If you want to eliminate alternate paths to answers, the sure way to do that is not say anything at all.
I believe the following are potentially bad consequences of Grayson's/Mikey's claim spreading:
- People work on insuring others, at the expense of other activities that they would otherwise believe to be more valuable.
- Insuring people (or the ACA) is deemed a failure because mortality rates do not come down as "expected", plausibly leading to the ACA's repeal.
- A developer expends time working on the project expecting mortality rates to improve; when it doesn't, the uncritical idealist becomes an uncritical cynic, rejecting any future promise of saving lives/improving things.
I believe the ACA should be understood to promise increased insurance rates leading to (a) less medical bankruptcy and (b) moderate improvement in certain healthcare measures (not mortality). I don't think it should be deemed a failure in any sense if it fails to reduce mortality amongst the newly insured.
Right. It sounds like you're implying that was his intent, and he failed. I really think it was an noble effort to get to the heart of what might be a misleading soundbite. Very much in the spirit of HN.
Is the spirit of HN nitpicking the smallest things?
Unless I'm misreading the polifact article you linked to in [2], it says that the 2009 study did control for smoking, and that they did a better job of controlling for such factors than previous studies.
> Still, their work stands out from previous efforts because it used more recent survey data and presented a more apples-to-apples analysis between the uninsured and insured populations. For example, it compared deaths rates for uninsured smokers with insured smokers, as well as other factors such as drinking, obesity, income and education.
This probably a significant exaggeration. It is based on a 2009 study[1] which examined correlation, not causation. It did not control for many factors that may be relevant (e.g. smoking). The study expressed this in much more careful words: "Lack of health insurance is associated with as many as 44,789 deaths..." (This number was then divided into 45M uninsured in 2009 to get 1 in 1000). Politifact did not rate this claim due to lack of information[2], but they previously rated "Half-true" a number half as big[3]. The latter essay cites work that did control for relevant indicators and found: "the risk of subsequent mortality is no different for uninsured respondents than for those covered by employer-sponsored group insurance."
I'm not sure where to place blame:
- On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?
- On Alan Grayson and similar folks, who are smart enough to understand the difference but are happy to assert causation?
- On Abbott, who implies causation, pointedly rejecting caveats ("it's not an exaggeration") in order to motivate developers?
I don't want to blame brandonb, particularly. I very much support his recruiting effort. In fact, I would say that the government probably has a disproportionate number of people who can resist unwarranted self-justifications. But I don't think a statistic like this should be left unchallenged on HN.
---
[1] http://www.pnhp.org/excessdeaths/health-insurance-and-mortal...
[2] http://www.politifact.com/truth-o-meter/article/2013/sep/06/...
[3] http://www.politifact.com/truth-o-meter/statements/2009/aug/...