People who "just want to feel in control" are, essentially, externalising the cost of their choices while having better options available to them. There's not an upside for anyone.
Whereas people who need to drive, where work couldn't be done at all if they didn't, would actually be more efficient if they had clearer roads to get around on. It would save them fuel costs and time.
Really though, I sense that your argument isn't in good faith. A nuanced look at this can easily see that some activities in a city require a vehicle to add value, and others don't. Lumping the two together and citing freedom (to externalise the cost of your choices on to others) is a little like using tradespeople as human shields.
Yes, for exactly the same reason as I'm willing to strip people of the freedom to practice shooting at elementary schools.
And at the end of the day, are you really willing to strip people of the freedom to drive through an urban area?