Now sometimes that is the only way that you can change things enough. Just rip the bandage off and damn all the hairs its going to pull out. I don't have a good analogy for the impact it had on them, perhaps if the government suddenly outlawed the use of cars or any form of non-public transportation it would throw a similarly sized wrench into the mix.
Slaves were freed. Infrastructure was destroyed. A quarter of the white male population was dead. These factors together led to a large drop in per capita income, but it's impossible to try and assign levels of responsibility to each one. [1]
It's also worth noting that the formalized system of slavery was replaced with a de-facto slavery system called sharecropping [2]. Sharecropping involved forced borrowing of work necessities at a high interest rates. It led many farmers, both black and white, into the grip of wage slavery [3].
1. http://en.wikipedia.org/wiki/Reconstruction_Era#Material_dev... 2. http://en.wikipedia.org/wiki/Sharecropping#United_States 3. http://en.wikipedia.org/wiki/Wage_slavery
Cherry-picked, out-of-context quote from wikipedia: "Though the arrangement protected sharecroppers from the negative effects of a bad crop, many sharecroppers (both black and white) were economically confined to serf-like conditions of poverty."