People might say "if you care about black children in poverty then what about white children in poverty" but people almost never say that to black people. It's always a statement of policy in general terms, not a targeted criticism to black proponents of a policy.

In other words, people might think that we as a society need to care about poor white kids as much as poor black kids, but nonetheless black people generally don't need to make excuses about caring more about black children than white children. Nobody accuses them of secretly thinking that blacks are better for doing so.