You can listen to an interview she does on econtalk -- interesting to learn more about the hidden biases.
http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...
+1000. That book should be required reading for anyone working in machine learning. Written by a former Wall Street quant who has the math down cold.
What she knows about rampant bias in allegedly politically agnostic machine learning circles is that the formulation and production of answers is trivial when compared to the formulation and production of questions.
Super-relevant to this thread is her work on recidivism risk scoring algos run on prisoners and defendants. The feedback loops that these algos spur are seriously damaging the lives of huge numbers of persons in the criminal justice system far beyond proportionality for the offenses that brought them there.
If you train an AI on data that solely consists of trading firms & banks, you should recognize that it's an AI biased towards detecting activity at trading firms & banks, and that it might be lacking in other areas.
It becomes dangerous when such an AI is marketed and assumed as unbiased and the source of truth for detecting all insider trading activity.
In other words if done decently well, they'd obtain data from all places but focus on the problematic areas till they reach equilibrium and then you redirect to the next hotspot, no?
It's a common known fact that crime-ridden, mostly minority neighborhoods have a complete dearth of services such as police, firefighters, and ambulance. Even back in the 80s, Public Enemy released a song called "911 is a Joke" because if you called 911 from a black neighborhood, they won't respond.
It's routinely known that cops will rather stay in rich neighborhoods and let the poorer neighborhoods fester. This happened during the LA riots where Koreantown burned because most of the cops went to Beverly Hill, etc, to protect the rich people.
I saw this first hand in Detroit about 8 years ago. I was driving through a bad neighborhood in Detroit with my friend, and we entered the neighborhood of Grosse Point, a rich area. Cops followed us until we left, which is their way of saying "you don't belong here". Meanwhile, among the burnt down houses and broken windows of Detroit proper, you couldn't see a single cop.
The difference? The neighboring town had a couple decades long reputation of being a hotbed for crime.
That reputation would be cemented in an AI system that was trained off of police data.
https://www.youtube.com/watch?v=MfThopD7L1Y
Basically, one instance of bias is the fact that many crime-prediction models are trained on police data, which means they will predict crime in places more often targeted by the police anyway. Then the model predictions even amplify that effect, since more training data may be generated from the places now more often policed, etc.
There's lots of resources out there on AI fairness these days. I think everyone who tries stuff like crime prediction should read up on that topic.