There is a strong consensus among economic historians that slavery didn't move the needle that much. Cotton production was perhaps 5% of GDP. The slave stock constituted perhaps 15% of national wealth.

https://sci-hub.tw/10.1016/j.eeh.2017.12.002