That's how a discussion starts being meaningless. Do you apply the same logic to everything else, at least in your argumentation?
Everything you do has the "millionth of a penny" influence on something but you will tend to discard those effects particularly because they're small enough to not be meaningful in any way. Except when you don't like the argument, and then the millionth become the cornerstone of your rebuttal.
How many times higher is the cost of other externalities, having the extra traffic in the city in terms of road fatalities, general health decline, increased maintenance needs, and so on? I'll bet it's not millionths it's millions. It's even more hidden and costs way more so why do you prefer it?
In my book, negative externalities are worse than taxes. If we agree that a car driving in Lower Manhattan inflicts a non-trivial cost on everyone else, then not taxing it leads to socially inefficient outcomes.
Effectively, locals are paying a price either way - either by having their bus moving slower, inhaling fumes, etc, or by buying goods and services that reflect congestion pricing. The difference is that congestion pricing aligns incentives - for instance, delivery drivers may choose to travel to Manhattan during the off-peak hours whereas in the status quo they do not care at all about inconveniencing others.
Thought experiment: If we could somehow ban private residential use of all cars and only allow "work" vehicles, there would be no congestion and no tax.
It would also increase the incentive for people to play games like claiming their personal car as a "work vehicle", throwing a little advertising decal on the side, things like that.