My guess why they don't use overbooking: it's really complicated. Both finding the right balance of overbooking as well as managing the procedures at the airport. Ryanair has simply decided in favor of simplicity forgoing some additional revenue

But risk would also be much higher - as you say, handling overbooking correctly is complicated, and when it goes wrong it can go very wrong. Ryanair didn't just renounce revenue, they renounced risk and the costs associated with managing that risk.

> And some of Ryanair's tickets are certainly below marginal costs

The operational word being some. Much-fanfared rock-bottom prices nowadays are very limited, and the cost of a seat goes up very very quickly after they're exhausted. The cheap tickets are now a marketing proposition for the real ones.

Edit: then again, with their relatively recent business plus fare (which allows ticket holders some flexibility), who knows...