Yes, manually driving a car will become illegal probably sooner than one expects.

While I'd like to leave the driving to Otto most of the time, sometimes I really like to drive myself, and would miss it.

I like my drag racing manual transmission and clutch, it's a heluva lot of fun to drive on the open road, but it's agony in heavy city traffic. (The engine just will not run at low speeds, the clutch is very grabby :) It can be really hard to get moving uphill from a standstill without rotating the tires in place a few times.)

There's no reason why manual driving needs to be made illegal.

A much more likely scenario is that safety features, starting with AEB collision avoidance, will become mandatory.

This will progress to the point where autopilot will always be running in a supervisory role, ready to take over in an instant if you're doing something dumb that could harm yourself or others.

It takes about 25 years to cycle the road-going fleet

Older cars can be prohibited from using, first, HOV-style "automatic" lanes, then highways in rush hour, then highways altogether, much sooner than that.