Those suits are so tight that after one (or a few) use they are effectively ruined. So it comes out to hundreds/thousands of dollars per race, not to mention during training.

Another thing to consider is that swimmers internationally compete on the same playing field — any aspiring swimmer can measure their times in direct comparison to the greats. In this way, even a child at his first swim meet is indirectly competing with their idols. I can understand why there is pushback against removing that aspect of the competition.