I would say relatively true of the southern colonies. New England, slavery import was banned rapidly, slavery itself banned fairly early (some states almost immediately) and it was arguably never a load bearing pillar. Virginia in particular and the southern colonies only avoided starvation by stumbling on tobacco.

I'd also note slavery was also influenced by how land distribution happened in the colonial era. Lands dispersed under more feudal models lent themselves more to slavery and indentured servitude. Lands that for various reasons that were rapidly sold were more likely to end in the hands of small holders without slaves or fewer slaves.