I don't really understand this argument. Surely reducing one of the operating costs of doing business to zero leads to economic growth right? In your hypothetical scenario, wouldn't the business owner just invest some of that profit into growing the business, thereby increasing productivity?

Alternatively, they can lower their prices in order to beat competitors. If they're producing a commodity, this can have the knock-on effect of lowering the price of finished goods using that commodity.

Another point of comparison is, in places where they discover "free resources" like petrochemicals, like Norway, that does lead to prosperity, doesn't it?

Sure, but imagine the alternative where the slaves are paid workers instead: then the same money that you use to invest into your business in slave scenario is instead spent by your workers on other businesses, which then use it to produce other things or invest in their businesses. Again, slavery, instead of increasing total wealth, just redistributes it in favor of slaveholders.

You are however averaging production to slaves too. So in countries where blacks outnumber whites you get much smaller wealth then what contemporaries commented on.

The alternative scenario is north in any case. The kind of society and production south was is plain impossible without slavery. It just did not exist.