How you think about exercise and time may influence whether driving increases your quality of life. To me, time spent driving is a pure waste with no redeeming value and I make every effort to minimize it. Whereas time spent engaged in physical activity (up to about an hour a day) isn't wasted at all since it improves my mood and energy levels. If that physical activity happens to save me a car trip then it's doubly useful.

For example, this morning I drove a kid to school because we were running behind and it was a ~10m round trip. Usually we bike and it's a ~20m round trip. I saved 10m of clock time but lost ~20m of exercise. Now I can choose whether to make up that exercise later and end up net ~10m poorer or skip today's exercise and end up slightly fatter and lazier.

(I agree about grocery shopping though.)