This assumes that the money has been earned completely fairly. Every law or norm restricting black people (redlining, restrictive covenants, discriminatory hiring practices, etc) is effectively a subsidy to white people. Therefore, some of the dollars in their pockets are stolen, in a moral sense. Restoring some of those dollars is only fair.

Mind you, I'm not sure if it's practical, but that's the logic.