The potential for abuse is not hypothetical.
While I was implementing medical information exchanges, every single participant considered patient data to be their own, to be used as they wish. Our (grand)parent company, a lab, was negotiating with Microsoft, Google, pharmas, etc. Each was trying to figure out how to monetize it. For example, targeted ads.
The C (executive) level players mocked HIPAA and the other (meager) patient and consumer protections the same way they mocked Sarbanes-Oxley, environmental protections, financial reporting requirements, etc. If you think Google and Facebook are bad...
---
My data, all that is known about me, is my identity. It's me.
At the very least, if someone's going to profit from my data, I want my cut.
>You're describing P-value hacking
Here's an example of what can happen when you take a huge corpus of data and throw an equally huge number of hypotheses at it to see what sticks: https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...
tl;dr: he "proved" chocolate causes weight loss by comparing chocolate- and non-chocolate-eaters on a very high number of health indicators.
That also introduces the multiple testing problem: https://www.wikiwand.com/en/Multiple_comparisons_problem
The more statistical tests you run against a set of data (EDIT: the more variables you test against a dataset), the higher the chance you get a statistically significant result from random error alone.
There are really three solutions to the problem of multiple comparisons: Either (1) you use a different threshold, (2) you use a different test, and/or (3) you correctly interpret that p=5% does not imply the effect is 95% likely.
There's absolutely nothing wrong with exploring a data set, as long as you are responsible in the conclusions you draw.
I really need to go back and study statistics, this is getting embarrassing.
I just found this on Google but the first page of this paper explains it a little better: http://www.stat.berkeley.edu/~mgoldman/Section0402.pdf
[1] Not really, but this is the cleanest way to sidestep multiple comparisons.
(1) First, you can certainly have confidence in hypotheses based off single data sets. If you have a dataset with 1 million hours of TV watching that show 0 correlation between watching golf and watching Judge Judy, it's fine to suspect there's little correlation. You don't need to run a second study to have an informed opinion.
(2) Second, collecting new data sets (or equivalently blinding yourself to partitions) doesn't 100% fix the problem either. If you test lots of hypotheses against your test set, then the odds that some of them are false rises too. Creating third- and fourth- and fifth-level validation sets just keeps pushing the problem up the ladder. In fact, there's no real difference between the requirement to experimentally validate results and the requirement to have a hypothesis 'work' on both halves of a partitioned dataset. The data doesn't care when you collected it.
Ultimately we just have to admit that tests based on randomness are sometimes randomly wrong. There is no perfect silver bullet solution.
This would be correct in the absence of investigator malfeasance. Unfortunately, investigator malfeasance is the problem we're trying to solve, so assuming it away is unwise. The requirement to collect new data imposes pretty strict limits on how many hypotheses you can test. The requirement to find a hypothesis along with a division of your existing data set such that the hypothesis holds in both halves is much more generous; it can be automated just as easily as finding a hypothesis that works in the unified data set can.
After you, the good guy, have specified which half of the data is the playground and which is the confirmatory test set, Evil Scientist can still run as many hypotheses as he feels like until he finds one that validates in both halves.
Under the rule "you can only validate a hypothesis by collecting a new data set dedicated to that hypothesis", we, the observers, have a way of guaranteeing that multiple comparisons did not occur. We have no such guarantee under the system you describe.
So to sum up: the rule I describe is not necessary in order to practice good statistics for your own benefit. But it is necessary in order to have a good statistical argument for convincing someone who can't directly perceive the contents of your mind. It's an auditing tool.
However, every test for a correlation against a data set has some chance of yielding a false positive or false negative. This chance is called the p-value, and typically .05, or 5%, is the minimum requirement to be considered "significant". But that means that if you test for 20 or so correlations, you would expect one of them to be wrong. And the only thing that can fix that is reproducing the test with a different data set.
Searching for "science reproduction crisis" will give a lot of good results for further reading.
This topic is also what this XKCD is about -- and it's not a coincidence that there are 20 "test" frames with a .05 p-value:
A p-value of 5% means that, IF the null hypothesis is true (IF!), then there's a 5% chance of getting results as extreme as measured.
A p-value of 5% does not mean than you should expect a rate of 5% false positives & negatives.
And, it looks like power is the error rate for false negatives:
https://en.m.wikipedia.org/wiki/Statistical_power
Too late to edit my original to fix this.
You're describing P-value hacking, thus named because hack scientists use this technique to publish papers about nonsense.
See for example: https://en.wikipedia.org/wiki/Genome-wide_association_study
There's a figure in there depicting associations with P-values of 1e-8: https://en.wikipedia.org/wiki/Genome-wide_association_study#...
[1] https://en.wikipedia.org/wiki/NHS_Connecting_for_Health [2] https://en.wikipedia.org/wiki/Health_Insurance_Portability_a... [3] https://en.wikipedia.org/wiki/Health_Information_Technology_... [4] https://en.wikipedia.org/wiki/Healthcare_in_Denmark#eHealth
I wouldn't underestimate the technological barriers to making interoperable health record systems actually useful. There are a lot of different kinds of medical information (SNOMED CT, the best ontology for healthcare, has >1M concepts!), and the best way to structure that information is an unsolved problem. There are lots of different ways out in the wild (complicated by there being lots of half-assed EHRs that were just made to grab incentive money), and the standards that are out there don't really help things (they are so broad that basically every EHR implements their own "flavor" of the standard).
In order to study something, he has to:
* Come up with a hypothesis that X may cause Y
* Request access to data about that hypothesis
* He is only given the data regarding his hypothesis
* He can then study whether his hypothesis has merit or not
We should be dumping these whole datasets into machine learning and having computers give us potential links to explore. Obviously there will be plenty of things that turn out to be unrelated, but it's also very likely the computer can find links that a human would not have considered.
I don't see it changing any time soon in the US, but I suspect other countries with this data will use it, and we'll find the next generation of medical breakthroughs no longer come from the US.