Oh wow. this happened to me recently. I foolishly believed that it was true since I've heard of companies like Reddit that have policies like these.
It isn't hard to write a comment about it that will show up in a search engine, and while that is far from perfect, it will allow employees who have freedom of motion to steer clear of slimy firms and those who don't to at least be able to negotiate from a slightly less-weak position.
One commonly mentioned alternative explanation is simply that people will quite reasonably not risk negotiating aggressively when they perceive their negotiating power as being lower. Given the number of women or non-Caucasian/Asian men who've reported being paid less or held to a higher standard than peers, I would seriously consider the possibility that the data is showing us a symptom rather than the cause of the pay-gap.
One of the problems is that when women lack such knowledge, they come across as less confident because, in the statistical sense of the word, they literally are less confident. In other words, in an absence of hard information, you cannot be sure if the figure you are asking for is too high, too low or just right. But when they evince this lack of confidence, people usually try to give them pep talks rather than hard data. It gets treated like a lack of self esteem rather than a lack of information.
I do think you are correct that women simply cannot afford to negotiate as aggressively and this is a factor. But I also think they lack access to knowledge that a lot of men are exposed to casually simply because they are male and this makes it easier for them to hang with the right crowd and get anecdotes, etc. And that piece is what I was trying to capture with the word savvy.
I am a woman. I know a lot about negotiating. I remain poorly equipped to negotiate on salary because I lack information. Having general savvy about negotiating technique is insufficient. You cannot play hard ball if you do not have hard information particular to the problem space in question.
1) I don't need a job. You'll pay me what I want or no deal. In those cases things like what this recruiter pulled won't work.
2) I need a job. In this case it doesn't really matter to me.
But they also can't keep bringing in people above a company's ask. It's a balance that has to be managed.
A lot of large companies have tiered salary levels that are not in any way tailored for the individual. In Germany, we have a Tarifvertrag in virtually every large company, which fixes the salary.
I had a similar experience looking for my first job in Chicago a few years back.
I met with a fairly large general recruiting firm in the area just so they could feel me out.
I named my expectation of 50-60k to which the recruiter replied.
"You know... when I moved to Chicago I was living on 18k"
Not exactly the type of response I expected.
Everything is negotiable, including when you hear things like that and "that's not our policy", etc.
Some recruiters are seriously sketchy. Someone I know joined a company where the recruiter told her that they give all their employees at that level this same salary so there's no room for negotiation unfortunately. After she joined, she asked what other people were making, and found out that even a new grad had higher compensation than she did despite the fact she had several years of experience already.
Apparently, they tried to do the same thing to that other employee, but that other employee had another offer and just said that he would take the other offer, to which the recruiter apparently responded, "Oh wait, I'm sorry actually I think we can do something about that. I was looking at the wrong sheet."