By saying, "Pushing to get more even representation is at odds with business success or product quality," which is often outright stated and even more often pretty directly implied, people are basically saying, "Those 'diverse' people are inherently poorer workers with worse quality output." That's the subtext, and that's what I think is kind of offensive about that line of argument.