Maybe this is a cultural difference, but it is perfectly acceptable in the US to wear swim or gym clothes on the way to your destination, so this is a non-factor. You’d wear a shirt with your swim clothes, so you’d need a bag, but you most likely need one anyways if you are carrying a phone, wearing shoes, etc.