The only real margin is the swap hardware and any structural binding parts designed to pass loads towards the battery frame as if it was built in part of the fixed frame. True, it makes no sense to design and include those parts when swap is not on the market, so it's a tradeoff. But is it a salient tradeoff, does it make the car with the built-in battery significantly better? Only very marginally I would say.

The same for aerodynamics, there is really only one place to put a large battery pack, it's not like you can gain lots of design liberty by putting an unconventional shape battery in the bumper or in the roof. Swappability of the underside of the car has no impact on aerodynamics.

There is only one place to put a large battery pack. However there are lots of places to put a smaller battery back. Or you can have a non-rectangular battery pack in that one place to better fit the room you have. Cars are about compromises, so a standard swapable battery pack will be smaller than the customized for the car one you can make. A tiny economy car doesn't have as much space as a large family car so plan is harmful to families that want to go on vacation as they have to stop more often.