I think leaving that out makes a lot of sense actually. You can't assume that the mix is 50-50 just because you don't know the data. Anecdotally, I know more women who use a male or gender-agnostic version of their name professionally than I do men. It may be more common in fields other than medicine, but for example we only know "JK Rowling" as "JK" because her publishers were worried that having an author with a feminine name would make it harder to sell the book to boys.
I don't assume that it's a 50/50 split. I assume it's somewhere between 0% female to 100% female. And it's a big sample. If a mix of (men and women) doesn't perform somewhere between the all-male and all-female groups, that's a big problem with their hypothesis.
They're saying something equivalent to Group A scores mean 90 and Group B scores mean 95. If you then take a very large set of people that is drawn from Group A and Group B with an unknown proportion and they score mean 85 or 100, that would be fairly devastating to the claim.
I'm not saying that this should be weighted into the main analysis. I'm saying they should test their hypothesis with this.
Re: (1), if the guidelines are correct and effective, I would expect women to score better because of the probability density of older men. That is, the average woman is younger and "closer to the data".
Re: (2), if experience is helpful, I would expect those with more experience to do better.
A priori, the person least likely to do well is a person who is far out from residency but who hasn't practiced a bunch. That person is more likely to be male than female (based on the above, basically, the historical consequence of systematic discrimination against women in medicine), and is less likely to have been trained to recognize symptoms uncommon in men.
In their tables, they describe "Physician experience control." I don't see a rigorous definition of this. They have a section in the supplement on variable definitions where they do not describe this. What does this mean? From Table S2, you can see that the men have accrued a mean of ~14 physician experience points, while women have accrued a ~mean of 10, a tremendous difference at this sample size. Does this score refer to (1) or does it refer to (2)? If it refers to (1), then I'm not surprised that men do worse. If it refers to (2), then this is all the more surprising.
Supplementary Table S2 also has incorrect labels (they seem to have duplicated female-physician / female-patient and left out the female-physician / male-patient). More confusingly, in the male-physician / male-patient section, the label appears to say that 10% of the male physicians are female. That can't be what they mean - but can anyone else interpret this? Edit -- I think this is referring to the average physician at that hospital. Not well named in the supplement, but that's OK.
Finally, 25% of their data is with physicians with names that preclude gender assignment by their algorithm. If we assume that to be a mix of men and women, that group should perform somewhere between the male and female physicians. Why exclude this data from the analysis, rather than analyze it to confirm the trend?
Study is here: http://www.pnas.org/content/pnas/early/2018/07/31/1800097115...
Supplement is here: http://www.pnas.org/content/pnas/suppl/2018/07/31/1800097115...