How do you think confronting people normally goes? Have you ever been in a fight? Even if you "win" a fight, you don't have anything to gain. Throwing punches can lead to infections and serious injuries (including death).

It's not cowardly to not respond to disrespect with formal means.