If those roads weren’t tolled they too would be full of traffic and it would still take you an inordinate amount of time to get there by that route.

Seems like your sense of restriction originates from the density of your city, not the rules of its roadways.