I suspect it works like any other sail. On an ordinary sailing boat you’re aiming to balance the aerodynamic and hydrodynamic forces as efficiently as possible, if you’re heeling over too much you can reduce the power produced by the sail by changing the sail’s angle to the wind, by reefing the sail (which reduces the sail area exposed to the wind) or by changing the angle of the boat to the wind (a less upwind course means less heel).

Sailing boats with a keel usually resist capsizing because as they heel over the keel is pushed to windward at a greater angle, increasing the righting force it provides to the hull. Additionally the greater angle of heel decreases the area of sail that’s being exposed to the wind, further increasing the overall effect of the righting force. It would be interesting to see how these can be retrofitted to an ordinary cargo ship though, sailing boats are optimised for a particular amount of heel whereas as far as I’m aware cargo ships generally aim to avoid heel altogether as far as possible with large ballast tanks where water is pumped around.

The sails automatically retract if the ship encounters a bridge or other structure it needs to pass under. Presumably they could be made to do so if weather jeopardizes the sails or the ship with sails deployed.

I think sails will tear and masts will break sooner than the wind would be able to tip the boats over.

You're right. Sail ships are impossible. Too tipy.

Sailing ships have keels.

The presence of a keel isn't necessary - just righting moment. If you look at most very large sailing ships, their keel isn't a large protruding fin keel like on racing boats like you're imagining. A cargo ship has enormous righting moment just by means of its ballast - the windage of their hulls alone is enormous.

I still don't think using traditional sails is practical though, and kites are a far more elegant and useful solution. They require far less cost and are easier to retrofit onto existing hulls, can be easily scaled up or down depending on wind conditions, and in the event of even catastrophic failure are pretty safe (i.e. if the kite line were to break). The biggest danger of them is the kite line falling in the water and getting fouled on the propellers, but this is a solvable problem.

The keel is also required to prevent sideways motion which would eat up any gains made by the sails pretty fast. But I have no idea how well a cargo ship keel works in that regard.