We all know the systems are biased, because we feed them biased information - Cathy ONeil did a great job on this with "Weapons of Math Destruction" What puzzles me is why he doesn't just provide the counterexample to scare the hell out of everyone - if he inverts the racial distributions in the training datasets, then you'd have a system where the shoe was on the other foot, and both argument and fuel for removing bias from the training sets - the recognition systems only know what they know, and they don't really understand that they might not know, they only know the confidence they have in knowing - hence this is my favorite happy place example of the problem --https://youtu.be/UFVB5rnqjyY