Of course it seems strange. The output of the scientific/academic community is often presented to people as being a source of truth and scientists do little to dissuade this. To learn that it's all rotten to the core is upsetting.
What I try to do when reading academic papers is, if there are multiple on the same topic, read them and try to understand the differences. If different papers come to the same conclusion, that should make you a lot more confident in the result. If there aren't many papers on the subject then you just need to understand that this is kind of a "best effort" thing that is more likely to be true than a random guess, but not certainly accurate.
Different labs also do different things - mine is currently reworking a paper that was basically done because something we did might matter, and my students have developing tests as part of their workload for developing code.
Unfortunately, you can't build a career out of doing QA for science. Nobody will fund you.
It's all about trust, really. Many people choose to trust state-of-the-art medical research when they do vaccinations on their kids; some do not. You can indeed choose not to trust us as a 'source of truth', and it seems a fair chunk of the US political landscape is being led down that path; we'll see where it goes.
The core principles of science are in conflict with natural human behavior (which is one reason it took so long to invent science), so saying “they’re only human” is really no excuse.
And believe me I hate being defeatist, it made me a lot angrier when I was younger and I consider it to be one of the things that stopped me trying to go down the academic route so far.
While coders (who aren't just being boosters for particular technologies) have to live with the fact that the world is run on mountains of bad code, scientists of anyone engaging with academia or journal papers has to sleep at night realising how much bad science is being done and how much the publication process is skewed to publishing and politics over quality or fact checking.
I can no more fix it than I can fix all software bugs or politics in big corporations...
I think you're making it to be a bigger deal than it actually is. Most of these things are just noise. We just ignore it. If someone rises to prominence with outrageously fake results, rest assured that they will get shot down quickly by concurrents (all the incentive to publish fake results lies within highly competitive fields). If you know what you're doing, and work in a lab where people know what they're doing too, you can still do pretty good science. I would guess than in all communities, academic or not, there's a nonzero amount of bogus, over-hype, nepotism, politics and what not, and a subset of people who actually get things done and make the whole field advance. After a while, the test of time truly determines what was actually useful from what is bunk. That's how it's always been.
> people in the community know and whisper among themselves
If no one else, tenured academics should be calling out bunk in their fields, or are failing, IMO.
Scientific papers are made public for a reason. The correct information should not be accessible only to a small circle of people in the know. If incorrect papers are out there, those who can correct them have a duty to do so.
Often it's not so much false as empty or useless. Nevertheless, since all people in a community know the values of individual papers, and these are the same people 'making progress' in that field, I don't see how science is so much harmed. I agree the situation is kinda ridiculous, but it's not a "sky is falling" situation either.
>If incorrect papers are out there, those who can correct them have a duty to do so.
Sure. How much are you willing to pay me for me to prioritize this 'duty' over the literal hundreds of duties I already have at my lab?
Better rewards for uncovering bad science keeps it from devolving into a political contest. If you're not seeing the right incentives in place, you're either missing something or you're seeing an opportunity to capture some value by implementing those incentives.
No. If you increase rewards for uncovering bad science, it will just lead to false accusations.
> If you're not seeing the right incentives in place
Some people think that the very existence of incentives is the problem here. https://hbr.org/1993/09/why-incentive-plans-cannot-work
Isn't that what this study suggests is already happening, in as much as false accusations are just more bad science?
> Some people think that the very existence of incentives is the problem here. https://hbr.org/1993/09/why-incentive-plans-cannot-work
This article is full of opaque generalizations about human behavior that more accurately describes the baser impulses of individuals attempting to game systems they don't understand (participants in psychological experiments) and don't so much reflect the actions of self-conscious professionals with any semblance of dedication to their fields.
It's Saturday morning (and I got students' papers to mark) :)
-I don't think you realize the step it takes to actually write to Nature and say: "Excuse me, that paper you published is wrong and you should retract it." You're liable to alienate all of the authors as well as piss off the editorial board for pointing out that they let a mistake slip through. You should be prepared to face vigorous backlash, and be completely confident in your own results if you don't want the repercussions to overwhelm you. Many communities related to specific fields are small and niche, so making enemies with one team often means that you've actually cut yourself off a good part of that community. You should be prepared for awkward moments in conferences when you meet each other, incendiary questions when giving talks and scathing "anonymous" peer-reviews for your future submitted work. It's far easier to assume good faith, give the authors the benefit of the doubt ("yeah the implementation is buggy but what software isn't? The main idea is probably sound") and move on.
-I don't know about other fields, but in mine, there's an unofficial accepted consensus on a set of very high-profile papers that happen to be either complete bunk or utterly useless. Again, we're talking about Science, Nature, etc. These papers made a lot of noise when they came out years ago, and now no one in their right mind would base their work on them. Again, you can't just knock on Science's editorial board and go "Excuse me, that hyped paper from a very big shot is useless and you shouldn't have published it", so it's just something people in the community know and whisper among themselves. I guess it may seem strange to outsiders though.