I used to think this too, since I tend to value interactions that range between autonomy and cooperation. Even where I like competition, it tends to exist within cooperative meta-boundaries.

Over time I've come to realize that not all people are the same, that there's a distribution of how agreeable / disagreeable people as well as how assertive. And it turns out on parts of that distribution there are people who enjoy not finding situations that benefit them but winning benefit and status from others.

I don't know whether that's "most" people. I'd guess not and that the distribution gets thinner the farther from the average disagreeableness and assertiveness you go. But it's enough people that I no longer discard adversarial values as a potential motive.