I was once peripherally involved in a conversation about ML algorithms being used to determine credit worthiness. One of the bankers explained that race as a factor is illegal, but the algo partially reduced down to a handful of heuristics -- eg that "last names ending in 'z' were less credit worthy" -- and that while these all roughly correlated to race, they were perfectly permissible since the algo was a black box & no one ever ran the correlations (conveniently). TLDR: sickening.