Re:AI Causes Real Harm. Let’s Focus on That over the End-of-...
by janalsncm on Friday August 11, 2023 @08:19PM
(#37095606)
A couple of reasons. Essentially you are asking “what is wrong with a perfect system” but the problem is no system is perfect, and we spend most of our time dealing with the imperfections.
In my previous jobs the consequences of me making a mistake were that a page didn’t load or some content was unavailable or in the worst case deleted. But a mistake in a loan decisioning system is likely to be 1) much harder to detect and therefore 2) much more likely to persist but most importantly 3) vastly more consequential.
Second, there is no way to determine from first principles whether someone will repay a loan. So you will need to use historical data. Loan data is very messy and biased due to cascading historical legal and quasi-legal policies. You may believe you can erase one feature from the training data (e.g. race) but if your model can infer race from other factors (which is likely because other features are not randomly distributed across populations) then you have the same problem you started with.
To take a concrete example, there was a man who was wrongfully accused of fraud. Then after he won his lawsuit, he was wrongfully accused trying to cash his damages check.
Some amount of racism, classism, and negligence went into that decision. That’s the bad part. But the good part is we could see what was happening. If instead the process was automated, it would be a lot harder to prove the same problem.
In my previous jobs the consequences of me making a mistake were that a page didn’t load or some content was unavailable or in the worst case deleted. But a mistake in a loan decisioning system is likely to be 1) much harder to detect and therefore 2) much more likely to persist but most importantly 3) vastly more consequential.
Second, there is no way to determine from first principles whether someone will repay a loan. So you will need to use historical data. Loan data is very messy and biased due to cascading historical legal and quasi-legal policies. You may believe you can erase one feature from the training data (e.g. race) but if your model can infer race from other factors (which is likely because other features are not randomly distributed across populations) then you have the same problem you started with.
To take a concrete example, there was a man who was wrongfully accused of fraud. Then after he won his lawsuit, he was wrongfully accused trying to cash his damages check.
https://abcnews.go.com/amp/US/black-man-sues-detroit-bank-al...
Some amount of racism, classism, and negligence went into that decision. That’s the bad part. But the good part is we could see what was happening. If instead the process was automated, it would be a lot harder to prove the same problem.