Sure, if we accept the idea that traffic congestion increases, but I don't think that it will be the biggest problem because with current forms of congestion it becomes frustrating because you have to pay attention to the road and operate the car, while alternatively with a SDC, that'll no longer be the case and can actively engage in productive activities (or not).

I suppose there are other arguments to this; Maybe time critical events such as emergency vehicles attempting to reach a destination, but there's probably a solution to this...