I don't see why we should assume that this would reflect reality.
If a law has racist roots, i.e. if it is written to target a particular ethnicity, then we should expect that certain ethnicities really do break that law more than others.
I get what you're getting at. If the data collection is biased, massage the actor on the data to compensate. Got it.
...but if you're uncomfortable with the consequences of unleashing a Machine Learned system trained on your data as part of your decision making process from the get go, why in the heck would you want to anyway? There is no benefit to doing so when you realize that the system only works as well and unbiased as the sum of the biases of the data architects, data populators, and data interpreters combined, and that extra loss factor you added is, in fact, another bias. An unacceptable on to the minds of many no matter how disagreeable one may personally find it.
Better to just leave it in human hands to deal with on a case by case basis by that point. At least by doing that we stay ultimately in control of the overall decision making process.
People being in the loop is not a bug. Trying to solve social/political problems with poorly thought out applications of inexplicable technological systems is.
You can add such a prior by adding a term to the loss function that penalizes any difference between the way the groups are treated. The math isn't hard, only the political decision of what is protected and what isn't.