You could do your own charging at home or at work otherwise.
It also hinders innovation. You can't produce batteries with innovative new form factors because nobody will be able to change them out at standard swap stations.
Finally, consumers would object to it. If I buy a brand new Model S I don't want to go to the swap station and give away my shiny new battery for someone's degraded high-mileage one that has an unknown probability of exploding or catching fire.
What I mean is, that problem is just one of thinking of it as purchasing a battery, which you wouldn't do if swapping them was common.
The battery is probably the most expensive component in the entire car, and is ultimately a consumable. People would forever be anxious about getting swapped out with a battery with more charge cycles on it than the one they had, ultimately affecting the potential lifespan/resale of the car.
Not an insurmountable problem, but a problem that adds a fair deal of complexity to managing the inventory at the charging stations. From what I’ve read, the software on the few Teslas that used this service didn’t understand the battery had been switched at all, and continued measuring battery health as if it was still the same original battery. I certainly wouldn’t have been keen on this idea at all unless the car correctly tracked these things.
But it’s also about about refueling (i.e. charging). Until a spot at a supercharger is as easy to come by as a gas station wherever you go, it won’t be the same. Not to mention having to wait 30+ minutes to charge, which can add up on a road trip.