You're correct that slavery held the country back. The Confederacy could not even produce shoes for their soldiers. The reason Lee was in Gettysburg was he was after the shoe factory in nearby Harrisburg.

The Civil War economically devastated and flattened the southern states. It's hard to see how the later wealth of the US economy was based on that.