So pay some trivial amount for insurance and let the insurance companies quantify the risk. Or refrain from overbooking. I see no problems here, just different solutions.

Insurance would be pointless for this problem. Insurers aren't magic risk-vanishers; they'd still specify an upper bound on their payout, leaving the airline with the unbounded amount, but the entire problem here is to upper-bound the downside!

That only leaves a) a law saturating the liability, or b) a contractual provision (and law validating such provisions) that saturates the liability. It still would explicitly prevent the possibility of colluders extracting an unbounded amount to be bumped.

You could buy an insurance policy that caps at, say, $100 billion, and then plan to go out of business if it somehow takes more than that to buy out someone's seat.

Or you could avoid the Rube Goldberg and just be upfront about the "buyout option"/"bid max" you're reserving on every ticket.

Or you could avoid that Rube Goldberg and just pay what it takes. The risk of a multi-million-dollar payout for a few oversold seats is lower than the risk of, say, being put out of business because your negligence caused one of your aircraft to crash into Manhattan and you end up being liable for tens of billions in damages.