If the stock market was falling/flat for a few years SWE comp would not be as high. By definition, most SWEs will not work at Netflix. The current situation of stable, well paying, and public tech companies is unlikely to persist indefinitely.
For every 20-30-40something FAANG engineer making $300-500k+, there are legions upon legions of SWEs working at IT-as-a-cost-center non-tech companies making a fraction of that amount.
There's also a career-long continuing education requirement that typically boils down to 15 hours/year of seminars or webcasts, all done on company time.
Even if you are fortunate not to have to do leetcode (i.e. frontend engineer interviews seem to diverge from leetcode problems nowadays), you still have to extracurricularly grind on coding in ways that you would typically not encounter during your normal work.
Software engineering is sort of a trump card in that it looks better than basically any other career on paper, but everyone can't be a software engineer. And while the actuarial field draws from a similar talent pool as software engineering, I think most people who enjoy one wouldn't enjoy the other - the former is much more of a business-y profession. I personally find the work I do as a SWE way less interesting than the work I did as an actuary, even though I'm not big on memorizing formulas.