For anyone interested, this concept is called https://en.m.wikipedia.org/wiki/Socially_optimal_firm_size
The thing that works is to have a coercive force which is primarily used to quell other large coercive forces. So you have a police force that prevents violence etc.
But it needs to be limited in scope, because it has no natural predators. It's always tempting to use the coercive force for <thing> where <thing> is literally anything. Laws are molasses. They have benefits, but they also have costs, and the fact that the costs and benefits apply to separate people makes it much too easy to do net-negative things.
Then someone would assume that power. Anarchy is not stable.
> Big companies could hire mercenaries to enforce their contracts
In the absence of any other government, any “company” is a government unto itself. And any that raises an army, by whatever means, pretty quickly makes that even more clear.
Notice how most of the squashing methods are government based: Patents, litigation cost of unmeritorious litigation, government-enforced contracts that should not be enforced under any reasonable antitrust rules, regulatory capture, etc.
The main non-governmental one is economies of scale, but that has a hard limit. The monopolist can't extract more from the market than the efficiency it creates by being a monopoly, or it would make it viable for competitors to enter. In an industry with low fixed costs, that doesn't buy you much.
The place where you would actually expect to see monopolies is the "natural monopoly" industries that are dominated by fixed costs. But even there, there's a limit to what the monopoly can do before competitors appear when there aren't government restrictions on entering the market. Starting a neighborhood ISP isn't cheap, but if the incumbent is offering slow service for high prices, it's what would happen, because paying (or even borrowing) $10,000 to invest in a new fast ISP that serves your own home is cheaper than paying more than that amount over time for worse service from the incumbent. But not if the incumbent has laws passed to make that purposely uneconomical or outright prohibited.