I was on my way starting masters on Mathematics when I discovered I would be a father, I had to find and start a career so that I could make ends meet.
From my perspective I am glad that happened, I was excessively ambitious, comparing myself with the great ones in the field, measured my self-worth on regard of my ability to solve problems.
Having to swallow my pride, learn to talk with other people as equals so that I could provide for my family made me a better person. I am not sure I would have done this breakthrough isolated on my intellectual bubble, not even talking about the difficulties of engaging with egos at the academia.
Sometimes I miss the excitement of learning something really difficult or solving some problem from a new perspective, but I am glad I am mentally healthier. I am not saying that it's not possible to be healthy and pursue a high level on Mathematics or other intellectually challenging field, but that can enable bad patterns for some people.
I have definitely seen that in other people when I was doing my physics undergrad. And to some extent, I probably experienced that myself.
I was on the road to starting a Cosmology PhD with incredibly high aspirations but my (then) girlfriend gave me an ultimatum: physics or me. To be fair, she was the primary bread winner and a graduate lifestyle would have been an even greater strain on the relationship. So, I opted for the latter and went into software. Also, like you, I have no regrets.
I still miss physics incredibly and, you’re right, there is something exceedingly sexy about solving those (including mathematics here) sorta problems. I sort of see it as a way to reach across generations and walk the same paths as the giants have. Anyway, I do find a great amount of joy in software problems, but nothing beats that first love.
This is something that I have observed in the sciences: a pernicious, unquestioned, and false great man theory of how science progresses (I wrote about it a little more here: http://madhadron.com/into_the_sciences-sample.html).
I got a PhD in theoretical particle physics in 1995. My thesis was about the supersymmetric flavor problem, where we tried to infer information about physics we couldn't measure based on the naturalness criteria she criticizes. My thoughts on pursing this: well, I left physics. I thought it was too hard to make any progress. I went into software. I don't have any regrets.
I shouldn't admit this but I occasionaly daydream in a Walter Mitty sort of way about doing great things. I have had a few physics ideas related to things like quantum gravity that I think about occasionally. Just a few months ago I was daydreaming about me pursuing one of these back in graduate school and it being very successfull. I intertrupted my daydream with a thought, "I'm glad that didn't happen. Then I might have stayed in Physics."
So that is my answer to the question about investing my own time in searching for new physics. None the less, I wouldn't say people are crazy because they think there is some value to pursuing physics as is currently being done.