Vox is an awful source.

In this case it's the source of nothing more than an explanation of the study which we are already discussing.

"An explanation" can be wildly misleading.

For example i might "explain" to you that clean code is about writing the least amount of code possible and you might start code golfing your production systems.

If you want to know what the paper says, read the paper. Journalists are not scientists, most of them do not have the necessary knowledge to understand academic papers, nor do they have an incentive for doing it well. They do have an incentive for generating clicks though, generally by twisting the truth to make things sound more interesting or provoking than they are.

Exactly. My guess is the person making statements about the study that aren't supported by it to begin with also just read some bad explanations of it. This is not worse. Criticising widespread faulty "interpretations" and pointing out it doesn't say what all articles say seems way more likely to get someone to look at the source (especially when engaged in online arguments) instead of using articles to back their claims up.

Vox? I would recommend seeking elsewhere for truth.

For example, after parsing through the ad hominem attacks and nonsense in that article, their main point is that the Bangladesh study found masks to be effective.

Except that study is junk and all the reported effects were found to be a result of researcher bias [1].

Vox also misrepresents the Danish study, which is probably the best study to date we have on masking effectiveness.

> Progress isn't a straight line.

Yes, but truth is most likely to be found in whatever facts are orthogonal to vox' narrative.

[1] https://trialsjournal.biomedcentral.com/articles/10.1186/s13...

I disagree with your assessment of Vox's article. It offers two big criticisms of the conclusions being drawn from the Cochrane review, neither of which you mentioned, namely:

1. Only two of the studies included are actually about COVID and masks; it's misleading to say, "oh, this big meta-review of all the data shows that they masks didn't do anything against COVID in the pandemic" when at best exactly one study out of the 78 included says that.

2. Even those two studies weren't about mask use but about mask policy. They're interesting from a public health perspective but don't tell us anything about the actual mechanics of whether masks work or not, which seems to be how (some) people are taking it.

It also doesn't speak well to the Cochrane review if they're including a study that you yourself argue is junk!

> parsing through the ad hominem attacks

https://en.wikipedia.org/wiki/Ad_hominem

> "Re-analysis on the statistical sampling biases of a mask promotion trial in Bangladesh: a statistical replication"

I'm not remotely qualified to have an opinion.

That said...

The open (public) process as well as the critics sharing their source code is just awesome.

https://trialsjournal.biomedcentral.com/articles/10.1186/s13...

https://github.com/mchikina/maskRCTnote

I share the reviewer's hope that authors of the original study will respond.

Cochrane revisit topics from time to time to update their reviews as new studies appear. The question of mask effectiveness was reviewed in the past also. There's an interview with one of the authors of this round's review here:

https://dailysceptic.org/2023/02/06/dr-carl-heneghan-intervi...

So, a Cochrane review is a study which synthesises all available studies – all that we can find or identity – on a particular topic. It follows a highly structured format and is always preceded by publication of a protocol. All this is to minimise the bias. Also, it is extensively transparent. In this case we are looking at about 300 pages of review. Now, the review called “Physical interventions to interrupt or reduce the spread of respiratory viruses” is called in code A122 for short and I will be using that acronym simply because it is just too long a title. So the protocol was first published in 2006 and then the first version was published in 2007, updated in 2009, 2010, 2011, and then 2020, so this 2023 is the fifth update of this review. And the reason why we update the reviews is they are soon out of date if we don’t do that, especially in some fast moving topics.

This update didn't change the conclusions from any of the prior reviews.

Because masks are so politicized there were numerous attacks on Cochrane this time around, though nobody cared in any of the previous rounds. The Cochrane authors are aware of all the criticisms, but there were no justifications found in any of them to alter the conclusions of the review or their procedures for doing them.

> This update didn't change the conclusions from any of the prior reviews. ... The Cochrane authors are aware of all the criticisms, but there were no justifications found in any of them to alter the conclusions of the review or their procedures for doing them.

True. But as noted elsethread, Cochrane is not responsible for others misinterpreting the conclusions.

"Statement on 'Physical interventions to interrupt or reduce the spread of respiratory viruses' review" https://www.cochrane.org/news/statement-physical-interventio...

"The original Plain Language Summary for this review stated that 'We are uncertain whether wearing masks or N95/P2 respirators helps to slow the spread of respiratory viruses based on the studies we assessed.' This wording was open to misinterpretation, for which we apologize. While scientific evidence is never immune to misinterpretation, we take responsibility for not making the wording clearer from the outset. We are engaging with the review authors with the aim of updating the Plain Language Summary and abstract to make clear that the review looked at whether interventions to promote mask wearing help to slow the spread of respiratory viruses."

> masks are so politicized

Indeed.

If science is going to be "self-correcting" then it has to make mistakes in the first place.

These mistakes will happen from the original scientists, they will happen at the stage of editorial boards, they will happen at peer review, they will happen if external third parties start systematically reviewing every RCT.

So Cochran must similarly be scrutinized for their errors, because they will be making them as well.

And that's even before we get to the political factors outside of science misinterpreting complex data for their own purposes...

Yes and:

It's wicked hard just to get reproducible results (one facet of the replication crisis). Much less the challenges you list.

Confusion and miscommunication is the norm. Rising above that take Real Effort™.

One of my formative experiences was on a team trying to adopt the processes from the book Applying Use Cases. So simple. Really What could be more simple?

We had shared purpose. We all read the book (among others). We discussed. We all thought we were good to go.

And then the wheels fell off once real work started. Turns out we didn't agree. On anything. What is "the system"? What level of abstraction are we working at? What does this line (points at diagram) here mean?

Writing this now, experiencing PTSD flashbacks, I can confidently say I would have never succeeded as a scientist.