Here's the problem: "Start with some selection of people chosen optimally for job performance...." That's the hiring equivalent of "assume a spherical cow." Sure, if your existing process optimally chooses people for job performance, then any move toward diversity will reduce quality. Likewise, if you have a spherical, frictionless cow, then its speed after descending a slope with a height of 100 meters will be 44.3m/s. Both statements are correct and equally applicable to the real world.
With regards to your second point, this kind of analysis does not require that the employees selected are in fact optimal. It only requires that they're the company's best guess given the inferences from the limited evidence available. Any move away from picking the best guesses will yield choices that are not the best guesses.
To get back to the example of purple people, you wouldn't see a company today which only accepts purple people. However, it's highly likely that people involved in hiring decisions have opinions of purple people, possibly subconsciously, which influences their evaluation of purple and non-purple candidates. If this skews their evaluation toward purple candidates, then pushing things back toward non-purple candidates will give you better results.
You're right, hiring in tech has gender bias, but right now that bias leans in women's favor at every step. Other commenters have posted research on this point, but I'll add my anonymous, unverifiable anecdotes to the pile. When I was on hiring committees at my old company, I watched us hire women who performed less well on our technical interviews than men we rejected, for the explicit reason of hiring more women. But I'm savvier than Damore, so I knew when to keep my opinions to myself about it.
And still there's fewer women in tech. There has to be other explanations.