Sure people living in the center of a city do not have many reasons to own cars, but they still often use taxi, and some want to have cars but can't afford to. And then some people do not use cars normally, but have to use them when need to transport something heavy, travel with children, or simply due to illness. So assuming everyone who doesn't own car wants them banned is too optimistic.
Disclaimer: i live in a city not in US, i do not own a car (although can afford it), and use uber occasionally.
Car owners are actually the minority in many large cities, especially in the dense parts. (>75% of all households in Manhattan don’t own a car according to https://blog.tstc.org/2017/04/21/car-free-new-york-city/)