Since this often comes up as a criticism of policies that eliminate subsidies, I want to emphasize your point that there are simple ways to counterbalance such a harm.
The easiest is you shave off some tax revenue from the wealthiest users/residents and redistribute exactly the amount back to the poor they would need to purchase the good as no-strings-attached cash. This makes poor people who choose to use the previously-subsidized good no worse off than before, but frees up those who never wanted to use it to find a different use of that money (e.g. public transit, or moving closer and paying higher rent but paying less in transportation).
Just pulling on my reply to same, varying your tax on a person's earnings is a much better way of means testing life than by separately means testing every tax.
But what's wrong with that? We all pay for national defence. We all pay for nationally funded arts and schooling and science. Even people without children or cancer or an interest in weird concept dance. We pay for this stuff because it makes our countries better, as a whole. Infrastructure is no different.
Even ignoring that, how do you issue this welfare to people who are paying this/these road use tax? Tax them less at the pump or the tolls? Who implements this?
And that still isn't fair. They're not paying for the roads they use. This money goes nationwide. Or does Tincanville (pop 30) have to shoulder the cost of their entire road network now?
Unless you're willing to take this to the absolute bureaucratic extreme, and make some people pay tons for low-use roads (which age naturally, regardless of use) an indirect tax of some description is the only way to keep the entire road network viable.
Some people don't have a problem with that. But pretending we're in a state with even a mildly socialist emphasis on taxes, taking most from income and redistributing across the whole system is a a much more efficient way of making sure the people who should pay more (I know, not everybody thinks that way) do pay more.
We can't pretend that rich people don't also benefit from GDP improvements. A healthy economy helps the land/home/business/etc owners most directly.
In the current system the direct tax revenue from transport doesn't cover its costs, so we're shuffling money around from other sectors. A lot of those other sectors probably benefit from the knock-on effects mentioned, so why not shift all the costs of transport into direct taxation?
Such a change would create a 1-1 mapping between the usage cost an individual/organization pays and the usage cost our entire economic system pays. Lining up selfish and collective interest usually improves efficiency.
Now, the price we pay for that efficiency coming from our current system is that the distribution of those costs changes. However, the cost is the same either way so now we're really just arguing redistributive policy.
I'm wary to advocate this position too strongly because if implemented in a vacuum it would be a very regressive tax (in the hurts poor people sense), but I would be very in favor of it with something counterbalancing that harm.