WWII offered a source of capital (people stashing their money into a neutral state, and many never asking for it back) and some industrialization, but the Switzerland we think of today is really a product of the late 50s/1960s.
Economic growth in europe only really took off to levels we think of as "today" after 30+ years of intense postwar growth. It was rich before that, but still very rural. Movies of the era are quite revealing.
I an genuinely curious if there is an argument against Switzerland et al being counterexamples to the colonialism/wealth argument. The strongest one I can think of would be that Switzerland et al benefitted from proximity to capital and to the new techniques discovered by capital.
https://www.nationmaster.com/country-info/stats/Economy/GDP-...
And conversely, all those fabulous silver mines didn't exactly leave the Spaniards with a head-start in modernisation.