The paper assumes independence between the gender of the applicant and the gender of the students of the professor, which isn't the case unless you hide the applicant in a room away from the current grad students.
Furthermore, the paper mentions that they used both Student's t-test and an ANOVA, both of which assume that the underlying data is normally distributed; there was no mention of a normality test anywhere in the paper. If the data is not normally distributed, which could be the case, then it's violating the normality assumption of both tests, which could potentially make them invalid.
Finally, are there other confounding factors in the study? For example, if the applications were sent in different batches, the time at which they arrived does have an influence. For example, a recent paper evaluated the decision of judges for parole and found out that judges were more lenient right after lunch and were much more strict right before[1].
As they say, further study is required.
I wish somebody woyuld at least look into possible reasons. Personally I suspect the elephant in the room is motherhood. If there is an x% chance for the recipient of your grant to quit the profession after a couple of years, it lowers the value of your investment quite a bit.
You think women can't be sexist against women? Of course they can be! A member of the marginalised group can easily continue that marginalisation. I watched a recent current affairs programme about same sex marriage, and they'd found a gay man to speak out against gay marriage.
What's interesting is when people do the opposite. When they A/B test the name. Same content. Half the people have male names, half female. Now, if gender doesn't matter, and people are judged purely on their achievements, and we are in a real meritocracy, then there should be no difference, right? If there a statistically significant difference in how people treat the male vs. female applicant, then there's some bias going on.
Turns out if you do that (e.g. http://blogs.scientificamerican.com/unofficial-prognosis/201... ), you get men getting more job offers and getting offered more money. And this is scientists judging applicants for a science job!
Something is wrong here....