You can be implicitly biased without being aware of it. This is true for both humans and algorithms.
Yes, this is exactly what everyone thinks happened.
Nobody thinks people at Goldman-Sachs wrote
if (applicant.gender == "F")
limit /= 20
somewhere in their algorithm.But regardless of how the thing happened, if millions of people are treated significantly differently for no reason other than their plumbing, that's a major problem. People have been talking about this "accidental proxy for gender" for years now; there's absolutely no excuse for no doing a basic sanity check to make sure that this kind of thing isn't happening.
edit: typo
Here you have a bank and a company known for secrecy.
But you acknowledge this;
> random CSRs don't get a pithy explanation
What the throwers of hissy fits are pointing to is that, by building blackboxes, you can get whatever result you want (which doesn't mean all results are expected) with plausible deniability.
As a side note, Idon't really understand the thread's appeal to credit scores here, considering that the TransUnion rating system is supposed to be the inferior one that Apple is looking to replace.
But most of all, I'm shocked by the number of people here outraged by Apple's behavior who will not even be considering switching off their iPhone (Twitter OP included, hell, he even posted a screenshot of his recurring TransUnion payment still being served via Apple Pay). I actually happen to agree with OP, but this blaise attitude towards real customer complaints has kept me from using Apple products for years.
Anecdotally I've been told that (British car) insurers don't care very much about your real actuarial risk, they're focused more on whether you'll actually pay their premiums. Specifically I was told that work to integrate with credit checking services was a priority whereas an integration with the UK Government's service which gives them access to driving offences and other records related to a driving license was back-burnered.
The reason I was given was that in practice they'd found if you require drivers to give their license details, it causes a big drop in purchases, if you make the license details _optional_ lots of people fill them in, and you can just give all those people a better price even though you don't use the details to actually check anything automatically.
There certainly is. Ignoring the anecdotal data of people replying to him who saw the same outcome, credit scores for women skew lower than for men.
One of many sources: https://www.federalreserve.gov/econres/notes/feds-notes/gend...
Which is crazy, because for most of my guy friends (myself included), saying that their significant others/women make significantly better financial decisions is putting it far too lightly. I basically wasn't making decisions at all (besides savings) until my s/o set me right.
This discrimination is tragic and disgusting.
Credit scores specifically correlate with default risk, not some abstracted measure of financial health.
https://realestate.usnews.com/real-estate/articles/the-rise-...
One of the statistics in it: Single women can only afford about 39% of homes. Single men can afford more than half.
Historically, couples with traditional marriages (primary breadwinner husband, wife whose primary responsibilities were women's work) often did not bother to put her name on a real estate transaction or car purchase. This substantially impairs a woman's ability to establish a credit record and reduces her legal claim to assets.
If you get divorced, the person with a real career and resume to match will likely continue having a good income. Someone who was a full-time wife and mom will face serious barriers to establishing a real career at all and may well remain poor for years to come.
I'm quite financially savvy. Financial savvy only goes so far. Ability to pay still matters and men are more likely to have that piece covered.
Since I believe that gender roles is about splitting benefit and responsibilities, I asked myself what the outcomes of a divorce should result in in gender stereotypical situation.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5992251/
A man who focus on the traditional gender role of breadwinner and a woman who focus on the traditional gender role of social management will results in decreased income for the woman and decreased social support for the man past divorce. This puts the woman at increased risk of poverty, while the man has an increased risk of loneliness.
There is also some implied finding that women who remarry has on average a decrease in house income, while men do not gain persistent increase in loneliness. I do not think there is a major mystery why that is considering differences in inter-sex competition for men and women.
In cases where the explanation is something like "No, seriously, she simply cannot afford it." efforts to root out presumed bias are not only unhelpful, they are actively counterproductive.
I need more income. Period. If you were to approve me for a mortgage this minute for the $550k building I desire, I simply cannot make the payments. It's not going to fix anything.
First, get me the money. That's the one thing no one on this planet seems seriously willing to help me with and there is always some BS excuse.
Very true, because they aren't as smart and they don't work as hard.
No, wait, sorry, it's actually because there are stereotypes about women that aren't even remotely true that are reinforced decades after they were found to be offensive through institutional sexism such as the gender pay gap and the glass ceiling.
So, maybe the lower credit scores simply reflect the deck stacked against women in the workplace.
But, no, that's not remotely the entire story. Other factors that help suppress female income:
1. They tend to spend far more time on unpaid activities ("women's work") than men.
2. When moving to a new city, a man is very likely moving to a new job. This usually entails a pay hike. A woman is much more likely to be following her man's career to a new town. This typically involves taking a pay cut and derailing her career aspirations.
3. Men are more likely to pick a college that supports their career goals. Women are much more likely to attend whatever happens to be locally available and affordable.
4. Men who start businesses are typically making a career move. Women who start businesses are more likely to be starting a "lifestyle business" to accommodate other demands placed on their time and energy, such as special needs children.
I'm sure I could go on. I've read research on the topic for literally decades to try to understand where my life went wrong, but it's ultimately pointless since I can't cite specific sources, most of the members of HN aren't familiar with such stats and, no, just taking my word for it as a SME in the sorts of things that have negatively impacted my life is simply not anything anyone here will do.
1) according to him, the bureau they were using for underwriting data indicated she had higher credit score
2) "conditioning on credit score which is correlated with gender" is not conditioning on gender. The algorithms that derive credit score from a set of credit data do not condition on gender.
It does if the algorithm is based on flawed assumptions, or worse, fed data based on institutionalized sexism.
Since no one is actually doing this, we're back to something like "maybe (altho there's no evidence for this either) people indistinguishable from Mrs. DHH actually do look more likely to default, which is society's fault and thus Goldman should just take the hit".
FTA:
“My belief isn’t there was some nefarious person wanting to discriminate. But that doesn’t matter. How do you know there isn’t an issue with the machine-learning algo when no one can explain how this decision was made?”
The anecdotal evidence, that many women are claiming that they received much lower credit limits than their partners, despite similar or superior credit conditions, would seem to indicate otherwise.
I don't believe anyone is saying that they're directly doing it. Just that their algorithms have that outcome.
Of course not, but why spoil the outrage mobs victimhood narrative du jour?
It could be "it's a new product, we randomly assign credit limits to see how it affects behavior".
It could be "it's a community property state and we're overexposed to this household if we give the second card the same limit as the first".
It could, realistically, be almost anything except for evil bankers deciding to use illegal criteria to underwrite that has a side effect of limiting the amount that can be charged to the account each month (you know, how they actually make money).
Oh, random CSRs don't get a pithy explanation of a multivariate nonlinear underwriting decision to poorly convey to customers? That's kind of precedented!