Energy: when you don't need to own your car, you can use the best car for the job. A commuting car doesn't need to be able to carry 5 people around. It can be much smaller. Again, the transition period likely won't look good.
Harsher economic times: this is good old technological employment, where productivity rises faster than demand. My 2 cents: self-driving cars will significantly contribute to technological unemployment, and that's good, provided we manage it well. An obvious measure would be to generalize a 32 hours work-week, over 4 days. It would mean less unemployment and less overwork. Workload will reduce anyway. We might as well share this reduction, instead of giving it all to the unemployed.
It would work just fine (as long as the rates were adaptable). But that is orthogonal to whether people will appreciate it.
Perhaps we live in different worlds, in my world self-driving cars will create more traffic jam, lead to higher road-use taxes and to harsher economic times due to people in trucking/delivery/cab driving losing jobs.