Right, but then the end user is paying for it in proportion to how much they benefit. Someone who uses the roads more (directly or indirectly) pays more than someone who uses the roads less. Which is how it ought to be.

The argument that you benefit indirectly could be applied to almost anything. You benefit from the computers used by the businesses you patronize, so should we levy a tax to pay for them? Probably not.