Also, if you haven't been following the 'X-ray Carrots' fiasco, that is a BIG deal in science and Science. VERY short summery: Andrew Wansink at Cornell made up a bunch of data and results, he published them, he got caught. One of these papers was about calling carrots as 'x-ray carrots' makes kids eat more of them, hence the name of the fiasco. Here's a good way into the mess [1]. Anyway, some papers were retracted, there was an investigation, yadda yadda. Now, in 'normal' science, he would have been banished from the field for this nonsense. But in 'finding-or-famine', he got off essentially scot-free. Nothing happened, really. He's still the head of the department (AFAIK), he still brings in cash, the FDA still is using his findings. This is a huge blow to science and a huge boon to Science.
The existential crisis is now out in the open: you can cheat with impunity, publish whatever, if you get caught nothing will happen, just bring in the dollars.
Now is not a good time to go to grad school. I think they're just going to have to throw out ALL US based bio-science (at least) from about 2003 to whenever heads start rolling. It's all crap, maybe, who knows.
[0] Science and science are different, though look similar. Big Science is the grant machine and the big money side of science. Splashy, barely replicated, large clean buildings, pensions, perfectly fit for corporate needs. Think Big Tobacco, BPA plastics stuff, power poses like Cuddy's, that kinda stuff. It's like the difference between the Military Industrial Complex and actually kicking down doors in Mosul. 'Little s' science is what you are probably thinking of when you think of science generally.
[1]http://andrewgelman.com/2017/04/06/dear-cornell-university-p...
In my experience, about a third or half of the 'real' scientists I've met are firmly in "Big S" science. It's really not that different. People that complain about this a lot are often jealous, in my experience. The failures are certainly high-profile, though.
As for reproducibility, it's all case by case. We've (real? scientists) been having the same discussions about data quality forever. It just seems to have found a fresh audience. There's a reason a curmudgeonly PI might insist in Western Blots and - gasp! - Northerns from time to time. Most fields were built off of some real science. Molecular biology certainly was, and to think that just stopped happening in the early 2000s is the height of ignorance.
There is cause for alarm. I give pretty much all 'genomic' work a very wide berth unless the lab has a real genetics background, and aside for a few labs, anything with 'microbiome' in it is certifiable garbage. And clearly some entire fields are on shaky ground, namely psychology. But lets not circlejerk our pure 'little s' science ego's too vigorously.
You misunderstand, perhaps. I mean that the data itself is fraudulent, let alone the interpretations or the controls on the experiment. Many PIs have been making up the data, essentially. Pulled from thin air with a little rand() thrown on top. Now the flood gates to this behavior are open and 'accepted', at least by the people that cut the checks.
I think you missed a decimal point or two here -- $42,000/month for a 100 square foot office can't be right :).
Or you could get a large sample size to bring the p down to 0.01 or whatever. You know, do science.
The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.)
To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.
Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it.
There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
1. A legal framework for health-related products & services that meaningfully protects individual privacy, and yet also preserves the conditions for the kind of velocity and rate of change that brings technological progress to the masses reliably and quickly.
2. Massive academic funding for more “pure” research in hardware & software design, engineering, the philosophy of what’s possible, the phenomenon and ethics of what we choose to build, and so on. TLDR Pay more people more money to do important research in the public domain, which I can’t help but think would attract many of the CS types you mentioned.
That exists, but there's upfront educational costs that need to be handled first.
> Massive academic funding
Look at Europe: their acedemics work much more closely with industry. Get a someone who can code in a room with a physician, and I garuntee they can find a problem to work on. Show a successful prototype, and we're way below "minimum viable product" here. Show a graph with real data and money will start looking for you.
Computer scientists, mathematicians, physicists, and engineers need to go to biology conferences. And biologists should be going to math, ML, and software conferences.
I still don't understand this one. The null model is (almost) always false. In this situation a small p-value is only a matter of sample size and measurement precision. How can it possibly indicate you should work more on something?
One thing I think may make sense is to get a bunch of p-values and then choose the lowest 10 or whatever to focus on for further work. There is no reason to include an arbitrary cut-off here though.
Also noticing this now... No, this is completely untrue. The p-value is not an error rate.
It's entirely reasonable, IMO, to do more "exploratory" studies at p = 0.05, and then do a follow-up on promising studies at p = 0.01
There will never be an objective measure for conclusion 'correctness'.
It will always be the case that you actually have to think about the data and statistics critically and decide whether the conclusions are appropriate. In certain well-defined situations, you can start to use harder metrics, but that's rarely where the controversy is in the first place.
So if you want to, you know, do science, I urge you to use that big beautiful brain of yours. As a bonus, you don't even need to commit mass rat genocide to do it.
We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?
This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.)
As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.
Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.
Are you aware of Registered Reports? https://cos.io/rr/
http://www.frankmcsherry.org/assets/COST.pdf
the paper you referenced?
Literally the papers are "I built a thing, and it gets faster when you do this thing to it." There's no condition on whether the base case is realistic.
To be clear, I don't think the authors of the papers he's taking down are bad people, or maliciously trying to get publications for uninteresting work, or too stupid to implement things efficiently on a single core, or anything. I think they're genuinely interested in the work they're doing, but responding to the pressures of the environment that allows them to keep doing that work.
There were (are?) enough degrees of freedom in the field in every study to always meet this threshold.
(Also, as a side note, I'd fully expect that CS is at least as vulnerable to the same thing. McSherry et al.'s COST work is a good example of something that seems far worse than p-hacking: instead of having a clearly defined hypothesis and publishing the 1-in-20 papers that falsely state it's true, as in other fields, there's nothing clearly defined in scalable systems and 20 out of 20 people who build something excessively complicated can get a paper out of it.)