Maybe, maybe not. Are there victimless crimes? A lot of people may think insurance fraud is such a thing but clearly that's a narrow view of "victimless". I do think you're right at some level or else being an asshole wouldn't have any survival advantage and would perhaps not have survived as a trait (assuming some heritable basis for being an asshole). But then how do you explain people who seemingly derive joy from pure trolling? No personal benefit other than lulz. Sounds a little sociopathic no? Then again, maybe these people are nature's chaos monkyes. Sowing the seeds of destructive change. Who knows.

I just know that I personally choose to try and not be an asshole.