So the net effect of that secrecy was that instead of the Nazis getting that money, the Swiss did.
Secrecy isn't agnostic. The lack of scrutiny enables--even encourages--people to do things they otherwise never would.
What's more the purported need for secrecy is often used to justify unethical behaviour from those providing secrecy to their clients: banks in Antigua taking 15-20% commissions on money transfers they all but know to be drug-related, keeping the assets of those that died in the Holocaust and so forth.
Those people now have a profit motive for such behaviour.
Now I'm not arguing there shouldn't be any secrecy but the fact that people with access to such information will feel morally compelled (or simply begrudged) to reveal information on those who are doing something shady behind the veil of secrecy gives me faith that the system will ultimately balance itself out.
Secrecy doesn't (and shouldn't) mean immunity from accountability.
It's this same reason that allows us to dismiss such laughable conspiracy theories as the faked Moon landings, aliens at Roswell and so on.
Let's not forget that many use these Swiss bank accounts for practices that vary from the questionable (eg evading taxes illegally) to the nefarious (eg hiding assets for the Nazis and various dictators).
[1]: http://abcnews.go.com/Blotter/t/story?id=5378080&page=1