Yes, the Dutch and later the English mostly got rich off of capitalism and trade (and early industrialization) at home.
Portugal and Spain had huge colonial empires, but that did not drive any industrialization or really long term wealth explosion like the capitalism and the industrial revolution did.
Is there any evidence to support this? I remember reading that exploitive practices like slavery added very little to Europe's GDP. There seems to be processes - like advancing textile production and coal mining technology and metallurgy - that began long before the age of colonialism, and thus would likely have continued even in the absence of the slave trade.