The next group will want more than $5 million, and so on. If the lottery didnt allow advertising of big wins that were made, a lot less people would buy lottery tickets.

No, there's a hard upper limit on ransoms; the cost of recovery.

What about when the cost of having the data exposed to the public is higher than that of recovery

Yeah, I was pushing that all under "recovery." Say it all sums to $C.

Arguably the bigger problem is you don't know that the ransomer will actually give you a valid key, but suppose you guess a likelihood P that they do.

Now you have some scenarios:

1. Don't pay. We're out $C.

2. Do pay, and get a valid key. We're out $R.

3. Do pay, and get no key. We're out $R + $C.

So the limit is at scenario 1 being equal to the combination of 2 and 3.

Set C = PR + (1-P)(R + C), and your max ransom R = CP

(You could probably work in additional costs for cleaning up even if the ransom is paid.)

I mean them publishing your data not you getting it back

That requires you to have that kind of data. The company could have be operating legally and not have compromising stuff. The ransomware team gains nothing if a company refuses to pay and has everything to lose by hacking. If there price is to high they are taking on a lot of risk for no reason. Hacks are smart people (I find breaking the law to be a bad decision but if one does it knowing the consequences and mitigations then they aren't dumb just unethical)