Because the cost is not an issue for higher income people. The poor either sacrifice something else to pay the toll, or they take a (likely longer, slower, or more congested) alternate route to avoid the tolls. This ends up costing them more time, which of course is a fixed quantity per day, so they again end up sacrificing. In a way it's regressive even if they avoid it.