By the way, this might not be true in the US, but in other places there's usually some sort of shared transit. Even if it's not a city bus or a ride-sharing network, there's usually some person with an unofficial "taxi" that drives others if you give them a call, or someone with a passenger van that makes a weekly trip to a city for a few regulars.
Old and poor people everywhere can't drive cars, so these solutions are widespread, you just need to ask around (calling the local shops usually does the trick).
I actually have moral objections to both Lyft and Uber. Other than that I only take vacations in places where public transit is an option. For more rural vacations, I usually just ride my bike there. Plenty of fun places to bike to from where I live (Portland, OR).
Otherwise, yes, someone else does the driving. My partner and I go out of town by car once or twice a year and she drives.
It's a tradeoff, and obviously if you want to go places where there's no public transport options at all on a regular basis, it's probably a good idea to at least have a license, but for my part that's simply been enough of a reason to bother getting a license.