When you say you're "against racial diversity" you are saying you prefer racial homogeneity. If that's not what you mean, you should rethink how you state your preference. Many people would take that statement to mean that you're an avowed racist.
The problem is that the word "diversity" doesn't actually mean merely having the presence of different races. Nobody who argues for "racial diversity" ceases to argue the moment the first black or Asian person is hired.
Rather, the phrase "racial diversity" has become code for its own kind of racism against white people. It doesn't mean the dictionary definition of diversity, consisting of multiple types. It means specifically eliminating and pushing out white people on the grounds that there are lots of them around in western countries, so harming them in some concrete, objective way isn't really harmful.
Moreover, the tactics normally used to obtain this so-called diversity are usually anti-meritocratic: literally the promotion and rewarding of people who do not deserve it on the basis of their work or skills alone, but just on the basis of skin colour. This is poisonous and demeaning to those people who do work hard, but don't benefit from being born "diverse".
It's really quite sad that this is actually a topic for debate, and that the ever more extreme elements in California have tried to make "meritocracy" a dirty word. But ultimately it'll be to the benefit of companies in other parts of the world who don't care about this strange and poisonous offshoot of political dialogue. I doubt there are many companies in Russia or China that force all employees to spend time on unconscious bias training, or who reliability promote unqualified people because of their DNA.
I would expect you to agree with this since you say you haven't observed a systematic difference in performance between these groups. No?
My own thinking is that it's good to invite a diverse applicant pool, but final hiring decisions shouldn't use diversity except as a tie-breaker. At the same time one should try to be aware of one's own biases, and rigorously careful not to devalue candidates unlike oneself.
In over two decades of work in corporate as well as entrepreneurial environments, I've not seen a difference in efficacy within groups that would be attributed to being either racially diverse or non-diverse.
For example, if there were three groups working on a given technical project, one composed of all white men, one composed of all Asian women, and one composed of a mix of race and gender, would the diverse group produce a superior result? My experience is that they would not. Sufficiently and equally incentivized and qualified, all groups would likely produce similar results.
So, I'd consider myself as somebody who's "against racial diversity" mainly because I haven't experienced it improving the core competencies of my company -- which is what I care about. Nor have I found compelling research supporting higher performance by more diverse groups. Thus, efforts to improve diversity, for diversity's sake, in my experience, feels like an effort to make a change that's not related, and might even be a distraction, to making my company more effective.