Governments that rely on violence may be perceived as legitimate by force but they frequently lack credibility.
Locke's _Two Treatises of Government_ is one of the earlier sources for this model, but it's a common theme in liberal political theory.
Not that I inherently oppose the use of violence to ensure that certain things that society has agreed on are enforced, Just that in many cases, I believe I would follow a current law set by a government even if there was no chance of punishment by the government (other than perhaps a public record)
(This is not to say that this is true of the majority of a given population, just that I believe that it is probably true about some people because I think it is probably true about me. (though I of course could be wrong in how I model myself) )
Note that I am claiming that there exist laws that I would follow without enforcement, but would not follow if they were not laws, but I am /not/ claiming that, for all laws, whether or not I follow said law is not affected by whether it is enforced.
A reasonable thought experiment is to imagine what would happen if the government convincingly announced that it would no longer perform any physical enforcement of any laws. I for one wouldn't immediately go out and murder or burgle anyone, because my reasons for not doing those things are firstly my ethical intuition that such an act is wrong and secondly the fact that people are likely to fight back. But you better believe that I would immediately disregard some laws, particularly ones prohibiting so-called victimless crimes.
I would make essentially the exact opposite argument as you. I think that most people believe that government is not primarily in power because of violence, I think they're wrong, and I think history (ancient to modern) makes it extremely clear that they're wrong.