The thing that works is to have a coercive force which is primarily used to quell other large coercive forces. So you have a police force that prevents violence etc.
But it needs to be limited in scope, because it has no natural predators. It's always tempting to use the coercive force for <thing> where <thing> is literally anything. Laws are molasses. They have benefits, but they also have costs, and the fact that the costs and benefits apply to separate people makes it much too easy to do net-negative things.
Then someone would assume that power. Anarchy is not stable.
> Big companies could hire mercenaries to enforce their contracts
In the absence of any other government, any “company” is a government unto itself. And any that raises an army, by whatever means, pretty quickly makes that even more clear.