Prior to the civil war, slaves were the single largest asset, by value, that America held -- more than it's land or agricultural exports; more than it's railroads, banks, or factories. Cotton was the number one export across all of America, not just the south, at that time. And it's not like this value was destroyed by the civil war -- post-slavery institutions were extraordinarily effective at restoring or keeping pre-slavery conditions long after the end of the civil war. The slaves may not have physically built most of America -- but it's silly to suggest America's wealth isn't largely, perhaps primarily, born from the fruits of slavery.

For sources, feel free to examine any modern history book.