Re:Facebook wants to redline your friends list
by pdkl95 on Friday September 04, 2015 @06:33AM
(#10169619)
> Facebook has really, really good people working on machine learning.
Being good at machine learning does not imply they are also good evaluating how their ideas will impact society, nor does it imply they don't have other motives (intentionally or not).
Unfortunately, too many people only consider the intended results when they create something, and rarely consider the larger impact their creation would have on other aspects of society.
> more accurate
It's questionable if this is "accuracy". They are doing very precise calculations, but the result is still going to depend on the assumptions in the design (e.g. that you can infer anything about default risk from someone's online "friends"), and the quality of the input data.
However, those technical concerns are not the main problem, which is redlining. Laws are slow in responding to changes, which allowed a lot of racists to maintain de facto segregation by simply claiming that someone "didn't qualify" for some loan. This was easy, because the complex nature calculating someone's default risk makes it easy to hide something that "just happens" to target a certain group of people. This becomes even easier (and provides more cover) when it starts to feedback on itself: (i.e. you're a risk because you're from $bad_area, but you have to live in $bad_area because we denied loans in the past). I highly recommend reading Ta-Nehisi Coates essay that explains this in far more detail[1].
Using a social-media "friends list" to judge loan risk is data with a very poor signal-to-noise ratio. That same "friends list" is also data that can be easily map to race or class, just like "current address" did for the lenders that redlined segregated areas to limit "mud people"[2] to subprime "ghetto loans"[2]. Which of these sounds like a more likely cause? Even if the developers at Facebook believe this is about "more accurate loans", someone has other plans.
The complexity of modern technology has made it even easier to hide the real decision-making process, and machine learning techniques are the worst, where "I'm not denying you a loan, the Magic Machine is! (even though I designed it and decided which data it gets to see)". These tools can certainly be useful, but it is incredibly important that you're not just creating a proxy (both software and legal) that hides the underlying decision making process.
As you say, the government absolutely has a role here, but governments are slow to change. In the meantime, it is important for the people implementing to remember that code is law[3].
> actuaries already use all sorts of information
Which is why some of them have had to pay discrimination lawsuit settlements in the $100M-300M range in recent years, with more lawsuits pending.
Being good at machine learning does not imply they are also good evaluating how their ideas will impact society, nor does it imply they don't have other motives (intentionally or not).
Unfortunately, too many people only consider the intended results when they create something, and rarely consider the larger impact their creation would have on other aspects of society.
> more accurate
It's questionable if this is "accuracy". They are doing very precise calculations, but the result is still going to depend on the assumptions in the design (e.g. that you can infer anything about default risk from someone's online "friends"), and the quality of the input data.
However, those technical concerns are not the main problem, which is redlining. Laws are slow in responding to changes, which allowed a lot of racists to maintain de facto segregation by simply claiming that someone "didn't qualify" for some loan. This was easy, because the complex nature calculating someone's default risk makes it easy to hide something that "just happens" to target a certain group of people. This becomes even easier (and provides more cover) when it starts to feedback on itself: (i.e. you're a risk because you're from $bad_area, but you have to live in $bad_area because we denied loans in the past). I highly recommend reading Ta-Nehisi Coates essay that explains this in far more detail[1].
Using a social-media "friends list" to judge loan risk is data with a very poor signal-to-noise ratio. That same "friends list" is also data that can be easily map to race or class, just like "current address" did for the lenders that redlined segregated areas to limit "mud people"[2] to subprime "ghetto loans"[2]. Which of these sounds like a more likely cause? Even if the developers at Facebook believe this is about "more accurate loans", someone has other plans.
The complexity of modern technology has made it even easier to hide the real decision-making process, and machine learning techniques are the worst, where "I'm not denying you a loan, the Magic Machine is! (even though I designed it and decided which data it gets to see)". These tools can certainly be useful, but it is incredibly important that you're not just creating a proxy (both software and legal) that hides the underlying decision making process.
As you say, the government absolutely has a role here, but governments are slow to change. In the meantime, it is important for the people implementing to remember that code is law[3].
> actuaries already use all sorts of information
Which is why some of them have had to pay discrimination lawsuit settlements in the $100M-300M range in recent years, with more lawsuits pending.
[1] http://www.theatlantic.com/magazine/archive/2014/06/the-case...
[2] These terms are from the affidavit[1] of Wells Fargo loan officers
[3] https://en.wikipedia.org/wiki/Code_and_Other_Laws_of_Cybersp...