A lot of hard-asses delude themselves about how helpless they are against the forms of physical violence they are most likely to confront. It's simple survivorship bias. The people you deal with day to day can't get away with violence. The criminals who can are going to win, or at least put you in a situation with an unacceptable probability of permanent injury.

(I don't think you're a hard-ass, for what it's worth. You're just using one of their arguments!)

Your argument seems to be, "self defense doesn't always work, therefore it's useless."

My argument is, "physical self defense, delegation to authority, situational avoidance, other tactics, and the ability to choose which of these tactics to employ works better than arbitrarily restricting yourself to a single tactic, or arbitrarily ruling out a single tactic."

I wouldn't go toe to toe with a violent criminal if I had any better choice. But the bullies we face as kids in school halls do tend to yield rather easily to a bit of physical force.