There are some crimes, like drug use, that are indeed over-reported in some communities due to over-policing.
But there are other crimes, like homicide, that are nearly universally reported, regardless of where they are committed. And those crimes are much more frequent in minority-majority parts of the country.
Turning a blind eye to this problem is a disservice to those communities, because they are the same communities that are most commonly the victims of crimes.
In this case, the increased crime in these communities are not caused by their minority status, but rather from a multitude of other factors with historical and societal origins.
I'm not sure why causation vs. correlation is relevant here. When, say, someone is up for parole the judge will review their past criminal history, conduct in prison, whether or not they will have a support structure when they are released, etc. None of these factors are causal. A judge cannot point to a past crime, or to conduct in prison and say "this factor will cause you to reoffend". No, the decision is made based on factors that are correlated with higher rates of recidivism.
White Americans = 70% of pop. , $100 trillion in wealth.
African Americans = 14% of the pop, $2.9 trillion in wealth
African Americans in particular owned almost 10-12% of the land in this country (true wealth) and were promised more from the government in reparations (40 acres) not long ago, but discriminatory policies stripped that land away from them over 100 years.
Being in the USA for 150-200 years results in atleast 500k-1million $ in wealth purely due to land and home value appreciation.
The average black person has about ~500$ in wealth. This isn't a fluke, this was designed... This is a country that criminalizes being poor more and more in many ways, so we can fall into the trap of revictimizing the underclass we created, this time using algos, if we are not careful.
When a judge is evaluating whether or not to grant parole, is he or she looking for a causal factor that will directly cause the convict to re-offend or is the judge evaluating the convict's situation with factors that are correlated with re-offending?
Showing causation with whole-group statistics is very, very hard.
The majority of the $500k-1m baseline in white middle-upper class wealth is in homes and land handed down as inheritances, this is not debatable. That property allows a certain amount of leverage to invest in education, businesses etc...
The ben & jerry's founder talked about this in vivid detail on the campaign trail with Bernie Sanders. If he was black where he grew up, no GI bill = no cheap housing = no appreciation of property/land over his childhood = no financial leverage to build his company.
I can take risks and fail without going bankrupt thanks to familial wealth, that is a tremendous luxury not afforded to the group I'm mentioning.
(And in any case, my point was just that it wasn't shown as causation in the stats you cited)
You are correct on that point. More than 50% of the 100$ trillion is held by the top 10%.
On the other point, my point is the first $500k to 1million of wealth was due to inheritance... not the total wealth of the 1%.
Follow up: At one point, any white male could move westward and get free land (whether indegeous owned or not). 40-100 acres after a century+ of property value appreciation is quite a bit of wealth. The $500k-1m number I use as a baseline for white wealth is very conservative.
This reminds me of the argument for the UBI. Instead of having thousands of tiny chairty programs sprinkled all over society, why not let the economy make itself as efficient as possible and then hand out the charity in units of dollars?
More specifically, the current breed of machine learning is correlation engine, the only strength these networks have is finding correlations in the absence of context or explanation.
Sure... and what about the chronic lead exposure issues in many of these areas, the effects of long-term overpolicing of minor crimes on family stability, and the impact of (lack of) generational wealth and education from being used as de facto or literal slaves for generations?
The 'minority' part here is a correlative factor, not a causative one.
Honestly, my preferred method of solving this would be to train the algorithm on a data set with all of the forbidden values included along with anything else the creator feels relevant - zip code, income, familial status, favorite sport, education - and then when running in production, against real people, don't give it the restricted information. Yes, you could theoretically extract race, gender and other protected stats from the information the algorithm actually uses in prod - but it has no incentive to, since a less-noisy signal is already provided.
For instance, suppose the optimal algorithm for your data set is some linear function of X, Y and Z - let's say X+Y+Z to keep things simple. X,Y and Z are all normally distributed variables, mean of 0 and the same standard deviation. Y has a 0.5 correlation with X, and a -0.5 correlation with Z. If not provided Y, your algorithm might come up with 1.5X+0.5Z as an approximation - extracting a bit of the signal for Y from the things it does have access to. It's suboptimal, but better than just X+Z. Unfortunately, Y is verboten - we're not allowed to discriminate on it, and this approximation ends up with results that track Y. So instead we train with X, Y and Z as inputs, so the derived model is X+Y+Z - and we can drop Y from that model in production, leading to a model that (while less accurate) shouldn't unfairly track Y.
The problem is that the output of those algorithms is used to drive decision making that has the effect of maintaining the status quo, by removing opportunities to change it.
I think your comment shows that you understand this. You accept that a decision may be correct, when measured in totally cold and statistical terms. But such decisions would not "change the status quo" and that would be a problem.
But that position is a deeply political one. Why should decisions at banks, tech firms, or wherever be deliberately biased to change the status quo? It's social engineering, a field with a long and terrible track record of catastrophic failure. Failure both to actually change reality, and failure in terms of the resulting human cost.
Injecting bias into otherwise unbiased decisions by manipulating ML models, or by manipulating people (threatening them if they don't toe the line), is never a good thing.
The whole point of using the algorithm was to make sure personal biases aren't impacting the decision. If we're going to alter the algorithm because we don't like the result, then why are we bothering to use an algorithm in the first place? Just use a human to make the decision. At least in that scenario potential biases have an identifiable source, as opposed to an opaque program that may have been made by engineers that deliberately tuned it to avoid any disparities because they think any disparity in outcome is fundamentally problematic
Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world?
> Is the goal of the AI model to predict crime rates in a hypothetical world where everyone has equal rates of lead exposure? Or is the goal of the model to predict crime rates in the real world?
The goal is to use the results of a model for something (pet peeve, I hate the use of "AI" to describe what are usually pretty standard statistical or ML models). The model you create, and how you apply/interpret it, depend entirely on what you're actually trying to accomplish or change with the results.
Depending on what that is, the kind of "bias reflection" we're discussing is hugely problematic.
For example, crime rates are not equal between men and women. If we force our AI to assign equal risk of crime to men and women then we will have introduced a bias that either under predicts the rate of male crime, or over prdicts the rate of female crime.
What matters is: What are the outcomes and consequences of active systems, AI or not.
For instance: How do the algo cope with derivatives of its own output being fed into itself as input at a later stage?
The reality is, truth is relevant and sometimes the truth is inconvenient. Tech workers may want to build an AI that measures risk of recidivism that produces uniform risks across race and gender. But the truth is, rates of recidivism is not the same across all groups. If we produce the desired outcome of equal reporting of risk, then the consequence is that men have their risk underreported to put them on parity with women, or vice versa.
The justice system should absolutely not use this system for any purposes, since justice is based on the circumstances of the individual case in front of it, not the societal statistics which apply in the aggregate but may not apply to that specific case.
And if an activist engineer deliberately biased the model to avoid indicating disparities in crime, then we will have sabotaged police's abilities to allocated resources. Hence, why this assumption that disparate outcomes are indicative of a biased model is a problem
We already know where crime is occurring. We don't need an AI model for that.
People aren't arguing about not using biased data, they're arguing that the model needs to be designed and trained so that the bias in the data doesn't affect the predictions in the model. And yes, that means deliberately de-engineering bias out of the model, which may involve introducing a counter-bias.
For example, you and others kept bringing up race earlier as a legitimate bias for criminal profiling. But socioeconomic status is far more correlated with propensity to commit criminal acts than race. A model of crime in LA based on race, for example, would assume that people in Ladera Heights are just as likely to commit crime as people in South LA because they have the same race...but Ladera Heights has a fraction of the crime as South LA (and several times the average income). Similarly, you would expect South LA to have less crime than the largely Caucasian Joshua Tree or Fontana...but both cities have higher crime rates than South LA, and for a period were some of the most dangerous cities in California. (Joshua Tree was the inspiration for, and original setting of, Breaking Bad. Fontana used to be known as the Felony Flats.)
What I am saying is that pointing to disparities in the outcome of these models to claim that the models are biased is not, on it's own, a reasonable conclusion. As you point out, people in Joshua Tree and South LA have higher rates of crime than average. So if our model flags people from these areas as high risk more frequently than other places, is our algorithm biased? If we deliberately make the model to produce uniform results across different locations because an engineer feels that it's problematic to have a model that produces different results between different geographic cohorts, then have we mitigated bias? No, that engineer intentionally introduced our own bias to make the model adhere to his or her worldview.
[1] See the 1999 documentary Blast from the Past.
But I think we're diverging considerably from the original point: that forcing an AI to produce equal outcomes despite unequal behavior in the real world is not the elimination of bias, it's the deliberate introduction of bias. If we have an AI that predicts recidivism rates, and we engineer it to produce equal predicted rates across all groups despite different between rates groups in the real world then we are deliberate introducing bias. The truth, regrettable thought may be, is that a magical AI that operates with 100% accuracy - the only people who it flags would have re-offended - is going to produce disparities because recidivism rates are not equal.
The blindspots end up being extremely problematic.
Statisticians refer to this process as "controlling for confounding factors." What really matters is what questions you're asking. Data is too often abused, not always intentionally, by people with vague questions.
If you're going to use a model trained on simple, biased data, to get, say, insurance estimates for a for-profit company, the model will probably successfully increase profits, so it was a good model.
On the other hand, if you're going to use the same model to help with sentencing, where your goal is to see equality and justice, then the model will do very badly, since it will punish many people for the community/skin they happened to be born in.
For instance, men commit more crimes than women. If we are building an AI that predicts risk of committing crime (say, estimating rates of recidivism) and we forcibly make it report equal rates between men and women then we will be creating a discriminatory system because it will either under report the risk of men or over report the risk of women in order to achieve parity. Engineering parity of outcome in the model when the real world outcomes have disparities necessarily results in bias.
What a terrible example. There are many other features being omitted that would predict crime rates. If anything, this is an example of enforcing your own bias on the model by not including all relevant features.
Problems like violence have multiple causes, but are widely understood to be linked to problems of poverty, inequality, and marginalization. Violence is also self-perpetuating through social networks. When an incautious user of statistical tools fails to investigate the causal story behind such findings, they're going to get the wrong answer. When a policymaker acts on incomplete or misleading findings, they're going to make the problem worse.
*Unless you're arguing that it's the fact of being non-white which makes people more violent, in which case I have nothing to say to you.
This is not how I interpreted OPs comment at all. I just read it as a note that data shows correlation between minority communities and crime. Minority is a flexible, relative term, to my understanding - which, notably in recent years is frequently attributed to "non-white", but over time has been attributed to many different groups of people (caucasian as well).
Interpreting it as meaning "non-white" in this context is yet another good example of our predisposed biases and relativistic understandings at play, I suppose.
Not to harp on you - just noting the differences in our perceptions.
At the end of the day, there's enough real, vile bigotry out there in our society, I think it's important to be extra clear when discussing topics like this.
I agree, there is are many negative biases in the real world. More so, I'd argue, on the internet, where people feel safer at a distance to state disagreeable opinions.
It's worth noting, especially when we are discussing the nature of how an AI learns.
As I wrote in my original comment, there are potential justifications as to why we might want to introduce bias in our machine learning models to prevent the from identifying things we don't want. But the point remains: building a model that, for example, associates technology with men and women at equal rates despite the disparity that exists in the reality is not the elimination of bias. It's the deliberate introduction of bias to make the model produce a result that is in line with what we consider a more ideal worldview.
I would argue that the case is precisely the opposite. I was literally talking to someone from Colombia yesterday who was complaining to me about how all Venezuelans are lazy criminals that are stealing their jobs, when in an American context, people wouldn't even distinguish between them -- they'd both be 'hispanics' or 'latinos'.
That person took a difference caused by a temporary political and economic situation and applied it to a population of people as an inherent property. I think that is actually the way that _most_ people think, and to make people think otherwise requires a lot of education.
(Not saying wether it's "true", "false" or 78% likely here)
Many people have been repressed their whole lives. It doesn't get better by digging heels in, but sometimes compromise and not trying to win all wars, may turn out improved outcomes.
A big problem is the whole dull-minded "right vs left" duality. Tribal mentality that replaces actual issues and solutions.
With enough education, you can make most people think just about anything.
That is true for people in the western world that have been heavily dosed with the ideas coming out of the western social sciences in the past 100 years or so, and not really anyone anywhere else, ever.
You're assuming that that disparity is not itself a result of bias in any way.
Again, there may be justifications to deliberately introduce bias in the model to produce higher rates of association between women and technology than would normally occur if the creators of the AI so desire. But this is not elimination of bias, this is deliberate introduction of bias to make th model match the creators' worldviews.
This is incredibly reductive - we use models for MANY purposes - prediction, inference, description, evaluation. Until you outline a specific use case for the model, understand where the data its trained on comes from and how it may or may not reflect reality, and think about whether the phenomena which you're measuring are important/sufficient for your application you CANNOT possibly justify your broad claims.
Say you have an AI that predicts probability of re-offending for prospective parolees. If we observe that this system is flagging men as high risk more frequently than women, then this is not indicative of bias.
But!, one might say, men are socialized to be more violent and are more frequently recruited by gangs. These factors are being absorbed by the AI. Yes, they are, and this does not make the AI biased. The job of the AI to predict risk of re-offending in real world conditions. Not risk of re-offending in a hypothetical world where everyone has the same environment and experiences.
If we were to manually adjust the model to make it display equal risks for both men and women then that is not elimination of bias. It is deliberate introduction of bias to make the output fit our ideal worldview.
Already you've made one of the most common mistakes in this area: confusing the label for the latent truth. We don't measure "re-offending", we measure rearrest and re-incarceration, which as I've discussed elsewhere are biased (in the sense of "unjust").
> If we were to manually adjust the model to make it display equal risks for both men and women then that is not elimination of bias. It is deliberate introduction of bias to make the output fit our ideal worldview.
Again, you are using the word "bias" in a vague and meaningless way. In this context, our goals are constrained by the law - in the US we guarantee people certain rights, and prohibit certain kinds of discrimination. In this example, it would indeed be wrong (and illegal) to punish men more harshly just for being men, since gender is a protected class. In the same way, it would be wrong (and illegal) to punish black people more harshly just for being black. In my state, whites and blacks use cannabis at equivalent rates, and black people are arrested 8 times as often per capita - an algorithm looking at arrests will see that and make the problem worse.
You've staked out some sort of naiive and bizarre "AI Purity" stance which completely ignores how such models work, and which misunderstands how we use models to learn things and solve problems. You're also mixing up different definitions of "bias". Some facts which you might find interesting:
* Penalization/shrinkage, which is ubiquitous and incredibly useful in many predictive models, is the deliberate introduction of bias to improve prediction performance
* Adjusting for confounding in regression/classification type models is one way of accounting for or removing bias in a statistically rigorous way - as you (imprecisely and inaccurately) say, "modifying the model to make it display equal risks". This allows us to measure effects, conditional on known risks or existing patterns.
* All statistical and machine learning algorithms suffer from sampling bias, where the data we observe doesn't match the reality in important ways - a fact which you're completely ignoring. If a model reflects the bias in a bad sample, why on earth should we accept that?
If all it's doing is associating being a man with a higher risk factor without taking into account anything else in that man's life then, yes, it's being biased. "Being a man" in this situation is a correlation to, not casuation of, actual factors making a person more likely to commit crime.
This is a real problem that has been happening for years in Florida: the automated risk assessments used for a variety of decisions, including bail and sentencing factors, labels black people as inherently being more likely to reoffend than white people, even after controlling for actual crime rates and recidivism rates. https://www.propublica.org/article/machine-bias-risk-assessm...
Briefly, (and simplifying for clarity) it worked like this: the ML algorithm scored the risk of criminals reoffending. It gave a "high risk" score if someone was 80% likely to re-offend, and a "low risk" score if they had a 20% chance. Out of 100 blacks and 100 whites, here are the scores the ML algorithm gave, and how many ultimately did re-offend:
100 black criminals
* 50 "high risk", 40 re-offended
* 50 "low risk", 10 re-offended
100 white criminals
* 10 "high risk", 8 re-offended
* 90 "low risk", 18 re-offended
On the other hand, the failure mode disproportionately punished the black criminals: 10 were "high risk" who never re-offended, while only 2 white criminals were. Meanwhile, 18 "low risk" white criminals did re-offend while only 10 black criminals did. So the score was more strict for some unlucky blacks and more lenient for some lucky whites.
However, a key point is that because the underlying re-offense rates were different between the populations (50/100 for the black criminals, 26/100 for the white criminals), the algorithm could not have done otherwise. That is, given some n% re-offenders, if you have to fit them into 20% and 80% buckets, your "high risk" and "low risk" counts are mathematically fixed. In other words, it wasn't the ML by itself that was the problem, but the COMPAS score that it was trying to compute that had these issues inherent in it.
I think this is a good example where ML wasn't biased (at least, not anymore than reality), but where people were too eager to turn to it for a poorly considered project. By wrapping up important questions in high tech algorithms, it's too easy to fool yourself that what you're doing is the best thing you could be doing and miss problems in the fundamental framing.
2) The whole point of systems is to impact the world, not replicate already recorded history or escalate problems.
Can the ML algo predict what the outcomes and consequences of its outputs are, and assess that? No?
Would you prefer an outcome where half of the white men were also marked as high risk, but with 16 reoffending? (to give the low risk pools the same recidivism rate) Now instead of unjustly marking ten black men and two white men as high risk, you're doing so for thirty-four white men. Is that somehow less discriminatory?
Because in the end, the choices are "accept that there will be a correlation between the outputs and race", "use no system that produces estimates or imperfect outputs", or "explicitly discriminate based on race to remove the race-result correlation". That's it.
I agree that an AI shouldn't use race or gender as an input. But we should not surprised when AIs have disparities when predicting risk of committing crimes, when there are disparities in the rates of actually committing crimes.
Perhaps you think there are certain feedback loops that need to be broken (your model is usually a static representation of the reality), or perhaps you prefer low rate of false negatives / false positives, or perhaps if a model is uncertain you would like to defer to a human.
If a model predicts high probability of re-offence, you might decide to delay an action as an example, or re-examine the case in more detail.
I get a feeling that machine learning practitioners are not properly trained to recognize such subtleties, everything is a binary softmax output these days :)
EDIT: This seems to stem from the obsession with purely discriminative models that either directly model a binary function y = f(x) or model a function that returns some kind of score, which is later compared to a threshold y = f(x) > theta. Neither of these lend themselves nicely to this conceptual separation between modeling and decision making.
Just because bad data may tend (in hindsight) to yield inaccurate decisions, or decisions based on faulty data, doesn't promote data as the main driver behind sound decisions. That would just be another belief system, and we don't have all possible data or omniscience to prove it!
0: https://www.nytimes.com/2018/07/22/opinion/flint-lead-poison...
Not a rhetorical question. Reading through the comment thread I'm slowly shifting more towards being on the fence, as opposed to having a "don't hack the AI" position.
But rather than hacking the AI, why not just get rid of the AI? What is the point of the AI in the first place if you're going to hack it to get the results you want anyway?
This is not a meaningful phrase - it adds literally nothing to the conversation but confusion. To bend over backwards to give you a reasonable answer: when you're interested in conditional effects.
Let's say you're interested in the risk of cancer associated with alcohol consumption. People who drink some are often found to have lower cancer rates than people who don't drink at all. Reasonable models adjust for wealth/income - estimating the risk of cancer association with alcohol CONDITIONING ON wealth/income changes the picture; the positive association is clear.
Adjusting for confounding is "removing bias", changing the effect estimates.
In a predictive context, using ensembles, NNs etc, the problems don't go away, they're just sometimes harder to detect (and they're dressed up in sexy marketing-speak like "AI").
Repeat after me three times:
"AI is not a magic truth telling oracle" "AI is not a magic truth telling oracle" "AI is not a magic truth telling oracle"
Your cancer example doesn't answer the question, which is a pity because I mean it when I say it's not a rhetorical question. I honestly want to understand your point of view better. In the cancer example, we wouldn't go in and filter the training set to force the AI to think that rich people have higher cancer than they actually do, etc. But that's precisely what it seems like people want to happen with AI: they don't want us to condition on wealth/income, rather, they flat-out want us to filter the training set to force the AI to think group A and group B are equal when that's not what the data says.
A good example is the criminal justice system in the US. Minority communities are much more intensively policed, prosecuted, and jailed. A model will, without care, use race or poxies thereof to predict "criminality" without understanding or accounting for the bias inherent in the label. If that is used in policy, it runs the risk of amplifying existing social problems and injustices.
We alway have to ask: "bias for what?", or the conversation will be hopelessly confused.