There's different kinds of privilege of course. You can be rich but still discriminated, you can be white but poor. And the way you need to help people who are poor is different from the way you help people who suffer from discrimination.
If poor people have a hard time getting access to something because it's too expensive, and you want to help them, then that money shouldn't end up with rich people. If a demographic group is kept out of universities or other opportunities because of their caste or the colour of their skin, then you correct that by giving them better access, maybe through a quota or some other form of preferential treatment that compensates for the effects of discrimination. And that shouldn't apply to people who don't suffer from that discrimination.
You need to address the thing that's going wrong, and not something completely different. Rich black people don't need money, they need to not get arrested over nothing by the police.
When we have programs that are specifically designed to help people monetarily, such as financial aid at colleges or welfare, they consider people's economic situation but not their race.
If you took your last paragraph, and substituted racial groups for castes, I think it would be clear that it doesn't make sense?