I don't find this argument convincing. You could make the same argument when you see a parent physically hitting their child, that it's not anybody else's business, but most of the civilized world agrees that you should intervene, either directly or by contacting the authorities. The child is helpless to defend themselves. The same applies in many countries worldwide today. Even if the majority of the population wants a change of regime, coordinated military power held by a handful of individuals is more than enough to suppress any hope of that.

Using your analogy you are not someone who's seen anybody hitting their child, you're someone who's fine with a random neighbour doing justice because he tells you he was bad.