This is a fundamentally wrong rule because, as we have learned, researchers and reviewers basic competence has come into play repeatedly. The further you get from quantitative physics, the worse it gets.

That said, if you are going to criticize something based on the idea "this result doesn't agree with my preconcieved notions so I'm going to say that their sample size was too small, the effect size was too small, or they made some sort of incorrect calculation, and conclude the paper can't be trusted", you owe it to yourself to read the paper carefully enough that you can make the determination they made a major error.

Unlike most people, I read the conclusion, then the methods section. For most papers I can't get enough from the methods section to conclude that I trust the authors to make their claim, because most methods sections are missing major details on how the corrections/controls were made.

Should we also look at final code assuming the programmer and their code reviewers lack basic competence? Or do we assume they made a best effort given their abilities and may have still made mistakes?

I believe the latter, and the rule I mentioned also leads to that assumption. Otherwise, why read the studies at all? But that's not what people here do.

> missing major details on how the corrections/controls were made

I also make an assumption that some corrections/controls are so obvious within their field of study that they aren't worth documenting - that it's only "average joe" readers who assume they aren't being done.

It is, in a way, akin to asking why programmers didn't document how their JWT is secured.

I don't compare software engineers and scientists by the same metrics. The important thing about a code review request is that it contains everything required for me to reproduce the effects of the code change in my own hands. Science does not require that- instead, the methods is typically the "least descriptive and helpful representation of what was done" (especially in the case of very complex biological experiments) and can't be reproduced with nontrivial effort.

BTW I've definitely gone into code reviews assuming the programmer lacks basic competence and been right. Other times, I've had to rollback other people's code (or stop a rollout) because an ostensibly genius programmer who made a change (and got a review from a starry-eyed junior) wasn't using their basic competence or testing their change at all.

Bad example. While the tokens themselves can only be secured in a handful of ways, JWT auth implementation is notoriously tricky. Even the official docs of some well known packages promote insecure practices such as storing tokens in browser localStorage.

There is a big difference between "here is what I think the authors did wrong" and "I did not read the paper, but did they even account for <obvious thing>?".

The former is perfectly fine and encouraged. The latter is incredibly rude and derails discussions.

Relatedly, you took the rule incredibly literally just so you could critize a strawman...

> Unlike most people, I...

Yes, yes, you a very smart and everyone else, in particular me, is an idiot...

no, I'm not smart- I'm wise. And brutally honest. It's amazing how gullible most scientists are, reading the results and taking them at face value.

Yes it amazes me you haven't won multiple noble prizes yet

Considering the average quality of software, I'd say people in the tech industry are quite low in the pecking order of who can criticize whose professional competence...

As someone who reads a lot of published research, no, that is a terrible rule and a lot of garbage gets published. Both historically and now there is no shortage of bad science that gets published, or good science that gets really misreported.

If you don't trust studies at such a fundamental level, what is the value in reading and commenting on such threads?

Studies don't have value just because they are a study.

Put another way, just because Cosmo publishes an article doesn't mean it's true - but Cosmo does also publish true articles.

Plenty of studies are great and move our understanding forward. More studies answer a very narrow - but important - question. Some studies are just inconclusive, but have value too.

You should read studies. You should evaluate the quality of studies as part of integrating them and their conclusions. At the same time just because something was "a study" or got published doesn't make it true or good or useful, anymore than being written in blue ink makes something truthful.

The value of science is that it does not require faith.

Not OP but

If someone were advocating making policy decisions based on astrological charts, I would read and comment in such threads despite not trusting astrology on a fundamental level.

I personally would go quite as far as to say that purely observational studies have as much predictive value as astrology, but "observational studies are as reliable as astrology" is a better first-order approximation of my opinions than "observational studies are as reliable as large-n randomized controlled trials".

I don't know what value vorpalhex gets from doing so, but I got value from the gp comment.

A specific form of bad research which could be at play here: The researchers programmatically searched the space of possible control variables to include until they came up with a model which maximized the apparent effect size for coffee, so they could publish an interesting & widely cited paper that looked good on their resumes.

With N different control variables to either include or ignore, that's 2^N possible sets of control variables. Odds are decent at least one of those regressions has a large effect size for coffee.

I would trust this sort of research more if instead of publishing a particular set of control variables obtained by an unspecified method, the researchers chose 100 of those 2^N possible sets of control variables at random, then published the average effect size from the 100 resulting regressions. Ideally they would make the code to reproduce this average effect size publicly available, so anyone could easily replicate using another 100 randomly generated regressions.

> On Reddit's /r/science board, there's a base rule that "criticism of published work should assume basic competence of the researchers and reviewers".

Since most studies/publications cant be replicated, I'd say basic competence is not warranted.

I don't think it's questioning anyone's competence to suggest that a single scientific result might have confounding factors. This is one study done by one team. If the result was repeated in several studies, all coming at it with different viewpoints then I'd start to believe the result. In this particular case though, there seems to be new research coming out every few months saying coffee is good or bad for you. If the results look like that then it's hard to put much weight on any single one of them.

> I don't think it's questioning anyone's competence to suggest that a single scientific result might have confounding factors

Honestly - I do. Basic competence in creating a study would definitely include identifying confounding factors that are directly relevant to the study being conducted. It's a basic part of the scientific method - both in creating the hypothesis and creating a test against that hypothesis.

I don't think reddit and in particular /r/science should be held aloft as a beacon of reason and good thinking. The pseudo science and junk that is regularly upvoted there proves that they should more frequently question the basic competence of the researchers and reviewers.