Should the owner of a toll road have to pay money back to subsidize the costs of the roads which give access to their toll road? Vehicles need to be able to get on/off the road before tolls can be collected.
Or, if we apply your logic, then it is impossible for people to live in rural areas, because the cost of the roads is more than the local economy can bear.
Road networks are an infrastructure investment. Since they are networks, the value comes from the entire network. What is the value of a road with no on or off ramps?
Because the value is so diffisive over both space and time, the only effective way to account for and recoup the cost is through a mechanism that can take a share of the economic growth resulting from the network.
i.e. taxes.
In some countries, the majority of motorways are toll roads: https://about-france.com/geo/motorways.htm
So in the result, only those who use (or intend to use) the toll road pay, but without the toll booths that slow everything down. The price also happens to be more reasonable.
Uhm, what?
Toll roads are about as fair as it gets. The alternative is spreading the cost over all (national) drivers.
I hate toll roads too when I encounter them, but making non-users pick up the tab for users is the actual highway robbery.