Computers just made the enforcement easier and with less opportunities to break out of it - for example, a sympathetic public servant no longer has the power to make exceptions since "the computer won't allow it."
An example given was a state wanted to spend less on low income benefits. They setup an overly complex computer system that decides eligibility and requires participants in the program to fill out multi page forms 100% accurately on a regular basis or their benefits are suspended immediately. There was no way to appeal this decision, only to re-apply.
Employees of the state agency no longer had any insight into why a form was rejected, and could not speed up the process to resubmit it because the participant had long term medical issues and needed potentially lifesaving prescriptions filled. As a result, many people who would otherwise be eligible for benefits had them denied.
This lead to a marked decrease in spending on social services and was considered a win by the state.
Anyone going through this rigamarole, that's always my advice. Keep pressing, they'll eventually accept. It sucks, but you have to play the game.
That isn't to say that corrupt public servants don't exist, and that they don't look nearly identical if you were to take a brief glance of their actions without looking at their reasoning, but that doesn't mean they're the same people or causing the same end effect.
"Sympathetic" and "source of inequity" are subjective. Letting someone get extra points on their test because of their race is equitable to some, unequitable to others. Who decides?
Equity, by its very definition, is unequal. Every single thing done in the name of equity is unequal. On purpose. Because the "job" of equity is to right inequalities in the system.
In practice it creates more freedom, because users have no choice in their top authorities (I'd have to move to another state or country if I don't like the laws) but often have a lot of choice in bottom level authorities (if the guy at the DMV is mean I can go to a different location)
I'm not saying it's strictly good, but it's much more complex. IF you are someone who truly believes the top level policy is good and perfect and has no problems, then yes, any kind of bottom level control.
But if your are someone who is harmed by the top level policies, you now have an escape hatch by which a public servant can help you overcome this.
For example once in NY I tried to register a car I had bought out of state and one DMV worker told me I could not because the previous owner was named Johnathan Smith but had signed his name as John Smith. Ridiculous, but it's possible they were following the letter of the law and an automated system would have said the same.
I simply went to a different DMV where a different worker took a more favorable interpretation. If it had been a piece of software making this comparison I would have been screwed.
I owned a vehicle for 15 years and it broke down on me, as a result I dropped insurance on it as I wasn't sure if I wanted to keep it or not.
fast forward 3 months and it turned out to be a relatively cheap repair so I went to get it tagged (it had expired while uninsured) and the woman at the DMV absolutely hammered me on the paperwork I had to show her to the point that even the agent at the insurance company was confused about what she wanted.
I go to another location and lo-and-behold it's no longer a problem. Maybe she was in a bad mood, maybe she didn't like the way I looked. I have no idea, but the point you're making is absolutely relevant in day to day life.
And if I can pontificate slightly further, inequities exist in life, there is no way to completely remove them. Giving autonomy to low level people and squashing the ones that abuse that autonomy is absolutely better for people overall, even if it isn't perfect.
I'm making this number up, but I swear that 90% of the time when I come across a company that's absolutely floundering under the weight of its own poor decisions, it's because those decisions are enforced by software such that people CAN'T work around it.
What ends up happening is doing anything crosses 15 silo's and each of those silo's absolutely enforces it's authority such that no single person is able to bridge the gap between even two of them.
What makes it hard is that there are legitimate reasons why you don't want a single person being able to write code, push it to production, open firewalls, open access to databases, add ACL's, etc. Depending on the industry, the need for controls should exist, and because there's SOME legitimacy there it gets pushed across the threshold past reasonableness to absolute pain for everyone involved.
---
And if I may, that's where the most dangerous solutions lie. When there's a legitimate use of them but they can be abused.
Well, you can solve some things. Internet and zoom went some way to solving "how do we work in lockdowns?"
https://www.cureus.com/articles/107920-increased-incidence-o...
The real social problem we ought to be solving is stripping bureaucrats and politicians of their ability to impose lockdowns on us at all. And there is no technical solution for that.
That's not logical. There's no valid need for traffic, but I appreciate Google Maps traffic reports.
So yea, inequality is a social problem, but it is amplified by technical problems. We’ve seen this time and time again, the Internet at large being the most glaring example.
We shouldn’t ignore this amplification effect while we wait to solve the social problems. Furthermore, computers are often viewed as some unbiased decision machine, which makes the problems worse.
I'm not really sure exactly what you mean by "feed AI inputs like race and it will make racist decisions", as that's an absurdly general claim that doesn't even really say anything useful. If you mean the model reflects the underlying stats that correlate with race, class, gender, then sure, yes, I agree the predictor will do that. To say that it's racist is an entirely subjective claim.
THe point of insurance isn't to avoid social ills, but to statistically minimize the impact of risk on individuals (while making a tidy profit for the insurer).
I'm sure if black wall street wasn't razed to the ground by a racist white mob (among several other attempts to gain stability, wealth, etc. by black people being destroyed by racist white anger) then maybe the model wouldn't need to "reflect the underlying stats that correlate with race...". But those underlying stats, and those correlations, didn't just happen in a vacuum, hewn out of the aether like a magical and consequentless thing.
Because it's fucking unfair, and humans hate feeling like they've been wronged for unfair reasons they cannot control. How would you feel if your loan got denied, you asked why, and you found it it would have been approved if you had been born in April instead of June? Don't pretend you'd love it and smile. You'd rant about it and tell all your friends how bad that lender is.
If the lender rejects you because your debts are too high or your income too low or something, or the zip code of the home you're mortgaging, that makes more sense and feels more relevant and in your control, at least.
Yeah, it's nice from the point of view of the lender, but the cost is too high to society.
Society feels so strongly about this that (in the USA) we've even passed all kinds of laws about what pieces of information are absolutely banned from being used to determine who gets a job or who gets a home. If you are a landlord or hiring manager you have to be aware of them so you can at least pretend to comply.
Sometimes I wonder if AI is already here secretly. Half of my replies on hacker news seem to basically boil down to teaching what humans are actually like.
I mean, I pay more for car insurance because I am immutably male. I don't feel great about this, but assuming that males my age are statistically more likely to be in crashes this makes total sense.
Likewise, I don't feel great that even if I were really good at basketball I'd be way less likely to make the short list of an NBA recruiter, because statistically I'm not going to be as good as someone two feet taller. It may not feel great, but it still absolutely feels fair too.
If people born in April instead of June were actually statistically more likely to be in crashes, I don't see how this is any different or unfair.
Because it’s radically unfair to do that! You’re seriously suggesting that some people should be penalized for the day they were born?
Taking the point more seriously, it’s hard to believe that the risks for whatever you’re ensuring have populations of people each month that are so different that a random person from each has a risk associated with them that birth month, as a signal, isn’t likely to discriminate unfairly almost as much as it helps. Naively, it’s likely to unfairly penalize almost half the population. Maybe you’re prepared to accept that because, hey, it’s right over half the time, but tell that to the vast population you’re penalizing.
And some things just shouldn’t be included. Health insurance, for example: isn’t it unfair to include risk of cancer caused by genetic factors in premium calculations? I guess you could say that’s different because parents could control birth month, but, man, that just seems so uncaring to me.
In the race example, let’s say there is a discrepancy in the underlying stats now, but it’s caused by structural problems in society caused by racism in the past. By using that as a signal to discriminate now it makes it more unlikely that the problems will ever be resolved (example: Black people cant get loans as easily, so can’t start businesses as easily, so the wealth gap is reinforced). If you think that’s ethical I would suggest that you are considering statistics too much and people too little.
Genetic risk of cancer cannot be included in insurance decisions (excluding voluntary life insurance) because a law was passed (GINA) preventing it.
It's not the role of the loan industry to make it easier for black people who appear to be a higher statistical risk to get loans. That's the purpose of government- to make it illegal to include genetic history in life insurance models- provided that society believes in that principle.
Another way to say that - one that appears less uncaring to readers - would be “yes, including birth month or race in insurance risk calculations would be unfair+, so it should not be done. If necessary, the government should step in to prevent this”.
+ you may squabble with the use of ‘unfair’ here - obviously it is not ‘unfair’ in the stastical sense, but that’s not what the word generally means in spoken English. ‘Inequitable’ would be better, but for some reason ‘equity’ seems to be evolving to mean ‘at all costs, produces equality of outcome’ and so is a victim to the same problem in the opposite direction. I chose the more common word.
[edit: I used insurance here, sorry - the same would apply to loans I think]
I do agree that if you expect corporations to act morally, then it would be sensible for them to be more careful in understanding and correcting long-term social ills.
It is trivial to bake an irrelevant distinction into a naive ML driven process.
Even if we set ethics aside, this is still a terrible idea because credit risk analysis is highly regulated in most cases. There is enormous regulatory risk to using birth month in a credit risk model. Particularly because hen a jury asks "what's an alternative causal model that could explain the bank's incorporate of this data point into their models?" their answer is going to be "absolutely none".
Instead, they look at who was approved, and maybe whether they repaid their loan.
So if a human making loan decisions isn't keen on giving out loans in View Park-Windsor Hills, a model trained to produce human-level performance won't be keen on doing it either.
Historically, at least in the United States, loans were unavailable to PoC, especially black Americans. Districts in many American cities were "redlined"[1], that is to say certain districts of cities were deemed "unprofitable" for banks. Redlining was policy and was targeted towards discriminating against black Americans, who were often victims to predatory loans with unserviceable interest rates, which commonly resulted in defaults. The defaults caused worse scores for people in the neighborhood and created a positive feedback loop. Historical housing loan data (and insurance data) includes this data. People today are affected by this historical data and this absolutely must be taken into account by the developers of these models.
Because of the entangled nature of real data, dropping "race" as a feature for training wouldn't solve the problem. Factors like zip codes (think of the redlined districts) would also influence the outcome[2].
Creating models, whose output can impact people so profoundly, e.g. can Jane get health insurance, calls for more reflection than just "numbers correlate!".
[1]: https://www.smithsonianmag.com/history/how-federal-governmen...
[2]: https://www.educative.io/blog/racial-bias-machine-learning-a...
Political solutions are necessary, maybe something like subsidized loans for people from formerly redlined communities to purchase and restore homes, or start businesses. Urban planning projects, like increasing mixed-zoning, pedestrian traffic, and good public transportation would help keep money in the neighborhood. Then there is the question is how to deal with gentrification and increase quality of living in a community without displacing people from that community. It takes a team of experts from various fields, the community itself, and goal-oriented cooperation.
Holy cow, so literally a horoscope? Lol.
How often have you talked to someone at an institution about some egregious wrong, and they defend it as "that's our policy?"
Towards the beginning of the pandemic there was a lot of this sort of stuff, say: https://www.nature.com/articles/s42256-021-00338-7