It sounds like he's saying "people are more likely to intervene if they think they could take the other person in a fight". If you're afraid because of a size differential, you're more likely to stay silent (not "you" specifically, but the societal "you")