Very few people do this, that I know.
I think widespread electric vehicle adaption will happen due to price but you have a very valid point on recharge time.
Basically your car is unusable if low and being recharged and there is nothing you can do about it. Unless swapping heavy batteries becomes practical.
Teslas have a driving range of ~200 miles or something like that right? After driving 200 miles, most people are happy to stop and stretch their legs for 15-20 minutes. Tesla superchargers take about 30 minutes to charge your battery most of the way. Drive another 200 miles and you're ready for lunch or dinner, which could be a ~1 hour break, if you're not getting drive-thru and a chance for a much longer charge.
The only limiting factors I see are cost, the availability of charging options (and density of locations) and charging for apartment dwellers. Charge times are probably "good enough" already and likely to get better.
And governments will tax electric cars to make up for the loss of fuel duty.