I can already see how something like this will further increase inequality due to unaccounted biases in the training data, for example worse no emergency response in black neighborhoods because there is less data about them.

The question is whether this bias is better than just relying on government agencies, which we know are absolutely infested with racists and fascists.