I can't speak for others, but I'm more fit because I live near mountains and other outdoor activities. I don't choose to live near those things because I'm fit. The causation is mostly the opposite of what you're implying.

Getting out and into nature will change your lifestyle for the better, no matter what your current abilities. And there are plenty of ways to get out there that don't require any fitness level whatsoever. Sitting in a boat and fishing, RV camping...I spent much of the past few weeks hiking in the Sierras and most of the people in our campgrounds were retirees.