There's also the climate of the South to consider — until the invention of air-conditioning, factory work in the South would have been at a significant disadvantage. Malaria, too, was an issue within living memory.
People don't realise nowadays how very inhospitable warm climates can be.
The South didn't even have a shoe factory.
Really, how productive can a workforce be that can't read, that have to be forced to do anything, that will not contribute any improvements, that have to be guarded at all times, that will sabotage the work if they can get away with it, etc.?
How productive would you be if you were a slave?
Slavery is a terrible economic system when compared with free labor. It isn't remotely a surprise that free economies economically bury slave economies.
Investors tend to invest in things they know about and can keep an eye on, which tends to be local. Investment capital uniformly fleeing the slave states to the free ones suggests that investing in slave states was a bad investment.
In those days, different states were like different countries today.