I don't find this argument convincing. You could make the same argument when you see a parent physically hitting their child, that it's not anybody else's business, but most of the civilized world agrees that you should intervene, either directly or by contacting the authorities. The child is helpless to defend themselves. The same applies in many countries worldwide today. Even if the majority of the population wants a change of regime, coordinated military power held by a handful of individuals is more than enough to suppress any hope of that.