Keep in mind that Damore is a single person giving feedback to a diversity course. He doesn't have the resources, time, or mandate to survey Google's employees to test whether or not this general trend holds. Perhaps, before acting on Damore's recommendations it would be a good idea to do some tests on Google's employees to see if general trends hold true in their specific subset - but that is a really different claim than the initial claim that Damore was unscientific.
If Damore had written that smoking causes lung problems and therefore Google should start taking some actions to stop employees from smoking, it might be merited to do further tests to see if the link between smoking and lung problems holds true for Google employees as it does for people in general. It would not be merited to call Damore unscientific because he had not yet done that research prior to making his suggestion.
When thinking about this you should ask what your priors are. If neuroticism is widely, across ages and cultures, found to be higher in women and men, and it is, then wouldn't your prior, without a Google specific personality survey, be that Google women likely have higher neuroticism than men?
To me it seems like an impossibly high standard to expect Damore to do a personality survey of Google to figure out if this oft repeated neuroticism result also holds true for Google employees.