Secondary effects from widespread automobile ownership already removed most of its benefit for most people (typically they're net-negative, in fact) for day-to-day activities. Full self-driving would at least put normal people on similar footing to the few who still gain significant advantage from cars (mostly people who can pay someone else to drive them everywhere). Buuuuut you're right that total time in cars might increase, making things overall worse—you're not exactly free to do anything when you're in a car, even as a passenger.

("Seems impossible, how's that work?" take commuting time and time worked to pay for car, insurance, fuel, and maintenance. Compare worked-time it takes to pay for equivalents on a bicycle, plus bicycle commute time if you removed all the space you drive past that exists only or mainly because of cars [which is... a whole lot when you start analyzing it]. Do same for e.g. the grocery store and other places you regularly visit. You'll probably find that most people in your city—including, perhaps, you—must be losing time to their car, and likely a lot of it, when the point was to get places faster, i.e. to save time. Cars in an environment where lots of people own cars end up being beneficial mainly for the disabled and people who don't have to drive themselves, so can partially reclaim that time for other activities, i.e. the rich. The analysis just gets more lopsided the more complete you try to make it—added healthcare costs, more dedicated time spent exercising to make up for inactivity due to cars, infrastructure costs, et c, there's a lot more time and money that can plausibly be put under the "cost of widespread car ownership" column. And yes, I stole this analysis from Illich—I ran the numbers because it seemed crazy, but he's totally right)