The biggest issue with the memo is that James Damore attached politics to it. Basically a fair bit of the last part of the memo was an open accusation that Google is leftist and is "alienating conservatives", together with several standard tropes from the populist conservative realm that did not advance his argument very much ("universities are leftist" trope, a strangely oversimplified ding at communism / Marxism, mentioning "political correctness" and violent PC protesters, etc.)

The politics instantly put the memo squarely at one side of the current cultural / political wars. With the current binaries in American politics right now, such will never lead to sober discussion.