No he didn't. He said they were inherently less interested in STEM, and speculated about a few personality characteristics from psych research that might explain why, but all of that is irrelevant. Damore explicitly said that you can't judge individual competence from a probability distribution, even if the distribution of competence of each gender were different (which they largely are not).
Here's a broad overview of the literature covering what Damore got right and wrong: http://heterodoxacademy.org/the-google-memo-what-does-the-re...
Turns out, he was right that women seem to have different interests. I suggest reading about the things vs. people hypothesis. You can get more women into STEM subfields that deal with people if you highlight those aspects. Hiring quotas and some of the other measures Damore was arguing against would indeed have no effect on gender diversity given these facts.
The other is that is it often given such uncharitable descriptions that people who have read it simply can't believe the hate levied at it comes from actually having read the thing. I've actually talked this whole topic through for hours back and forth with someone who is very against the memo, but even they they would call it a sexist manifesto. That sort of thing comes across as off the cuff criticism by people who haven't read the thing.
If he's made direct statements rejecting biological determinism it should be easy enough to cite them — and it'd be useful to do so to avoid distraction when his supporters try to rehash the arguments in favor of that position.
In it, he argued that women were essentially worse at logical tasks than men, and further that this made them poor engineers.
That's not a diversity memo, that's a sexist screed.