To add to the argument about security, the average stock market return after inflation is around 7%. With investable assets of $1 million, that means you're being paid $70k/year, just for already being wealthy. I imagine that no matter how low stress a job can be, it can be even lower stress knowing that no matter what happens, even if you never work again, you would have sufficient income to live.

(This glosses over the variability of the market, but that just adjusts the amount of wealth needed relative to the yearly expenditures in order to continue indefinitely, and the principle still holds.)