Framing your competition as diversity hires (and contemplating the outsized concern that they have on your life) says more about than you think. You're the bottom of the barrel at what you do. "Diversity hires" don't compete at a high level.
I for one know that if my competition wins, whether they are a minority or not, they do it on skill. I've elevated myself out of the concerns of competing with diversity hires, because I'm great at what I do. Level up.
Where's your crocodile tears for Vietnamese kid from south central (where the Vietnamese in America are on the lower end of the economic scale) who is admitted to Harvard instead of a Chinese, Korean, Japanese, or Indian kid from a well to do family? Lets segment Asian Americans and see how you feel.
When opportunity is equally earned, we can then look to representation. Your argument is just an attempt to claim all spoils.
How would you know if a colleague of yours is a "diversity hire" without making assumptions? How do you know that you're just not as good? Because you don't think their work is high quality? Maybe you're a bad judge because your work isn't high quality. Or maybe you only see part of their work? Or maybe your pre-judgement has already doomed you to biased analysis of anything they do.
Or maybe if you're worrying about this you're just not very good.
How could it have no effect on our lives? It has effect when you compete against a diversity hire, it has effect when you have to work with a diversity hire, it has effect when your manager has to decide who to lay off when your coworker is a diversity hire.