I think the cheapest and easiest starting point would be to offer people a time guarantee if they book, and contract with cab companies to provide capacity.

E.g. a bus route near where I used to live was frequent enough that you'd usually want to rely on it, but sometimes buses would be full during rush hour. Buying extra buses and hiring more drivers to cover rush hour was prohibitively expensive, but renting cars to "mop up" when on occasion buses had to pass stops would cost a tiny fraction, and could sometimes even break even (e.g. 4 London bus tickets would covered the typical price for an Uber to the local station, where the bus usually emptied out quite well)

Reliably being picked up in a most 10 minutes vs. sometimes having to wait for 20-30 makes a big difference.

Even just letting people know how full the bus is, in advance, would help a lot with that decision to take a cab etc. There could easily be a map or list of the physical buses and how full they are.

If the bus is full then the transit agency needs to run more service. Unless this is a "short bus" or your fares are unreasonably low (free fares are bad for this reason) your bus is paying for itself and you can run more service on that route to capture even more people.

NYC has this. Bus locations and estimated number of passengers on board: https://bustime.mta.info/m/index?q=M5

The status quo in many cities is ~5x overprovisioning just in terms of capacity actively on the road at any given time, and way more than that if you count idle capacity. You could overprovision by a lot and still come out ahead.