Wealth is not the measure of all things and it's definitely not something that gets spread around evenly. There are winners and losers to mass immigration. Mass immigration also reduces the notion of there being a common culture and destiny which reduces the social capital and civic mindedness required for public programs to work.
According to those who oppose caps, instituting caps will cause a bunch of pain anyway. If the EU want to punish the swiss, the best thing to do would be to allow that pain, which the swiss have indisputably brought upon themselves.
The whole EFTA agreement only works because Swizerland etc. are relatively small compared to the rest of the EU. If everybody had the EFTA deal and there was no EU, the whole thing would go to shambles within a few decades.
With most countries with high amounts of immigrants being either quite poor or quite rich (especially those in Europe) and Switzerland obviously belonging to the rich European ones... it's not obvious what good could come from trying to leave this list of the richest, highest-immigration countries.