Part of being worry-free is knowing that even if you became unable to work you would not go hungry. 100k would do that for a while (unless you had medical expenses) but not forever. As a result, at some point you would need to worry.

My point in saying all of this isn't to depress you or wave money in your face or anything-- just to be clear, I don't have that kind of cash. It's just that the math says somewhere in the $1M to $10M range is what you need to be legitimately isolated from personal financial stress.