For human managers, this can be hard to prove, especially if they do not promote people very often. But for algorithms, it is much easier -- multiple people can inspect the source code for suspicious logic. And even if this is some sort of impenetrable AI model, you can still run it with different inputs, and check if the output changes if you change the input race.
When implicit biases find their way into algorithms, it is usually a specific edge case that’s a function of the training data (e.g. facial recognition preforming poorly on darker skinned people due to training set imbalance). The biggest case of implicit bias in algorithms I’m aware of is when the training set is the behavior of society as a whole, as with search engines: for example, women search less for C-level jobs and so are less likely to be shown postings for C-level jobs when doing job searches (thereby perpetuating a vicious cycle), or websites depicting Black teens are more likely to show them in a criminal context than White teens, so searches for “black teenager” show pictures of criminals whereas searches for “white teenager” do not.
These implicit biases are not encoded by the algorithm developer but rather by the dataset the algorithm is applied to.
In the case of automated firing, I don’t see how implicit bias can creep in if the metrics are strictly work-related (e.g. fraction of on-time deliveries). For the record, I do not agree with automating performance metrics, since they cannot account for nuance (e.g. delivery drivers assigned to gated communities have issues opening the gate and thus deliver fewer on-time packages, which is not accounted for in the algorithm.) However, this is not a form of demographic bias, explicit or implicit.
And implicit biais goes way beyond their training dataset. Did they put a clause "if pregnant, then...", likely not. However, what conditions did they use for testing this algorithm and ensure if it works? Testing is really much part of the process as well.
I agree in the case of things like facial recognition, but in the case of search results, Google can’t exactly choose whether websites it indexes present “Black teens” in a poor light and “White teens” in a good light. Perfectly unbiased algorithms trained on data produced by society will always exactly reflect the unfortunate biases of society. (Ironically, attempting to correct for society’s biases in the algorithm would by definition bias the algorithm itself!)
This is (likely) not the case with Amazon drivers, since the training data are (presumably) strictly job performance-related, e.g. fraction of on-time deliveries, or number of times a worker showed up late. And even if these happen to correlate to demographics (e.g. if Black workers lived further from the warehouse than White workers [or vice-versa] and were thus more likely to show up to work late), tracking employees based on their on-time performance is not bias on Amazon’s part, since a White worker who showed up late would be just as likely to be fired as a Black worker who showed up late. The fact that one demographic (by no fault of its own) happens to show up late more often is not bias on the part of Amazon.
al•go•rism /ˈælɡəɹɪzəm/
n.
1. Obsolete spelling of algorithm.
2. An irrational belief in the infallibility of algorithms.Now combine that fact with the demographics of Silicon Valley companies and you get a recipe for products and algorithms that are "fair" only if you match a set of implicit criteria.
None of this is intentionally malevolent, just a product of hubris and "move fast and break things".
[1] https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-te...
[1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...
Besides if the goal is "make an algorithm that only base its judgment on population studies" all human decision making algorithms will be the same, and you can endlessly cry wolf.
TL;DR: One of the factors for referring people to treatment was doctor visits over the last year. Makes sense; unwell people go to the doctor, so those going to the doctor more often are more likely to be unwell. But, of course, due to historical segregation, generational wealth, etc, black people on average tended to go to the doctor less. So even if they exhibited the same risks, symptoms, etc, as white people, because they saw a doctor about it less often the algorithm was less likely to refer them for treatment.