News for hackers. Stuff that matters.
Re:Will GDPR Make Machine Learning Illegal? by protomyth on Sunday March 18, 2018 @02:32PM (#16613447)
> only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason.

That doesn't sound exactly legal as the actual reason had to exist at the time of the decision and an after the fact reason would not be the truth.

Don't the ML models log why they made decisions in the first place?

Re:Will GDPR Make Machine Learning Illegal? by narrator on Sunday March 18, 2018 @02:41PM (#16613505)
The log is we multiplied an 80x25x3 matrix of your planet's photograph by 30 40x40 matrices of other numbers and then fed it through a neural net which consisted of adding and multiplying a few hundred other numbers and we got this number, which when run through a tanh function, classified your planet as uninhabitable space garbage. That's why the Vogon constructor fleet arrived and cleared it to make room for an intergalactic highway.
Re:Will GDPR Make Machine Learning Illegal? by dogecoinbase on Sunday March 18, 2018 @02:47PM (#16613528)
The point of interpretability in ML is that the "reasons" that model arrived at a decision/classification/etc are incomprehensible (and, given e.g. the existence of adversarial examples in image classification, this should be expected -- no reasonable answer exists for "why" the model thought this static was a cat).
Re:Will GDPR Make Machine Learning Illegal? by shawn-butler on Sunday March 18, 2018 @03:41PM (#16613850)
It's not magic. It amazes me how many "data scientists" can't explain the basic mathematics behind their modelling.

If the outcome of algorithm/model can't be explained by its regulators then it shouldn't be making any decisions of any significance or importance to human lives.

When engineers fail to address cracking in a bridge and people die when it falls, humans are held to account and explain why this decision was so. If machine learning statistical models can't do the same, they better not be used to scan any bridges for structural cracking where I live and vote.

Re:Will GDPR Make Machine Learning Illegal? by philipodonnell on Sunday March 18, 2018 @03:48PM (#16613884)
I have heard approaches described that train a simpler ML algorithm to predict why a complex ML algorithm made a decision. Decision-tree models predicting neural net models. Paypal or Square I believe.
Re:Will GDPR Make Machine Learning Illegal? by vinn124 on Sunday March 18, 2018 @05:26PM (#16614465)
how does this solve anything? if a simple decision tree could predict the outputs of more complex deep nets, why not use the decision tree in the first place? also, what do you do when a decision tree isnt powerful enough, as in the case of many interesting problems such as speech, computer vision, etc.
Re:Will GDPR Make Machine Learning Illegal? by philipodonnell on Monday March 19, 2018 @10:11AM (#16618429)
The uses cases for this were around fraud detection systems where you need to provide a reason for the flag.

I don't have to know the complexities of how all the factors are interacting within a neural net to be able to tell someone that if they increased their credit score by 100 points, they'd be approved.

Re:Will GDPR Make Machine Learning Illegal? by protomyth on Sunday March 18, 2018 @03:55PM (#16613924)
Hold up, data scientists don't know why their ML is deciding things and companies are using it to decide important things in people's lives? Sounds like it should be banned until companies can actually explain the decisions and some data scientists need to take an ethics class.
Re:Will GDPR Make Machine Learning Illegal? by bobcostas55 on Sunday March 18, 2018 @04:28PM (#16614120)
>until companies can actually explain the decisions

That's almost certainly never going to happen. Our most accurate models are unintelligible, and our most intelligible models are inaccurate. There's a trade-off here and without some sort of magic I don't see us transcending that trade-off.

Re:Will GDPR Make Machine Learning Illegal? by protomyth on Sunday March 18, 2018 @05:07PM (#16614340)
This is where I don't understand. If the data scientists don't know the why of the decision making and the software cannot explain itself then how do they understand that it is accurate?

Going further, how in the heck do they justify using something like this on credit applications or anything else that can adversely affect their fellow humans? The ethics of such a lack of knowledge should give pause and frankly I hope they get sued if this is the truth of the matter. That is monstrous.

Re:Will GDPR Make Machine Learning Illegal? by Aeolos on Sunday March 18, 2018 @07:21PM (#16615107)
> This is where I don't understand. If the data scientists don't know the why of the decision making and the software cannot explain itself then how do they understand that it is accurate?

They understand that by evaluating the model on a "test dataset" that is, hopefully, representative of the real-world. This does allow you can explain why a given model makes the decisions it makes - it only allows you to understand how well it performs on the data you feed it.

Using an inaccurate or biased model to detect assign credit ratings is indeed immoral. This is orthogonal to interpretability: you can have a model that is both inaccurate and interpretable (i.e. it's wrong and you can say why it's wrong); you can also have a model that is highly accurate but not interpretable (i.e. it works, but you cannot explain why).

Re:Will GDPR Make Machine Learning Illegal? by protomyth on Sunday March 18, 2018 @07:39PM (#16615190)
Ok, I get the training and verifying via test data sets, but I really don't get how a piece of software cannot be explained. It runs on a computer. It executes instructions that have to be visible in a trace. Someone had to write and debug the initial code. The data for the customer flows through the system, so how the heck can it be a black box? This is not mystical, its computer instructions. I can accept that data scientists have built software that doesn't output enough information to model the decisions, but I cannot accept a artificial system cannot be trace to show the decisions. If this is the type of software that is going to be running anything that can kill (e.g. autonomous cars) or fry (e.g. medical equipment) a human then I hope someone makes it illegal as heck until the model can be reversed in every case. How in the heck do developer / data scientists think they can get away with a black box with no accountability?
Re:Will GDPR Make Machine Learning Illegal? by bobcostas55 on Sunday March 18, 2018 @08:02PM (#16615276)
If a neural network has let's say 50 million parameters, it doesn't matter if you can trace any of the calculations, you're never going to actually understand how the model operates.

Sometimes it's possible to extract a sort of conceptual understanding in some cases, eg you might say that this layer performs edge detection or whatever. But that's not much of an explanation.

They do it because it works. You trust your Uber driver to get you to your destination even though you have no idea how his neurons do it. People are going to trust machine intelligence in the same way.

Re:Will GDPR Make Machine Learning Illegal? by protomyth on Sunday March 18, 2018 @10:53PM (#16615939)
People are going to trust machine intelligence in the same way.

Given the xenophobic history of the human race, I have some severe doubts about that. Frankly, the first data scientist who ends up in court because the automated car decided to kill some kid on the sidewalk[1] to keep the death count down is going to find out real quick what a jury thinks of machine intelligence.

Thinking about it, I can see the black box in a car having to record the last minute of instructions run by the car's cpu. The whole idea of unknowable isn't exactly going to sit well with the NTSB. I can truly see this if data scientists testify in a Congressional Hearing that they don't understand how their creation came to its decision.

I guess I am in awe of folks who do not have tools to figure out what model all the learning has built. Where is the DTrace for ML? If your neural network has 50 million parameters then how the heck did the data scientist have a data set to teach it from that is in anyway complete enough to trust it?

I can just not see how it ethically can be unleashed upon people in a final go / no go decision affecting people's lives or livelihood. After reading this thread, I now hope people who are rejected for loans ask for an actual explanation and the exact manner the decision for their rejection was reached.

1) https://www.usatoday.com/story/money/cars/2017/11/23/self-dr...

Re:Will GDPR Make Machine Learning Illegal? by sgt101 on Sunday March 18, 2018 @04:45PM (#16614216)
What's the uplift between your super accurate model and 1-r? What's the uplift vs a 50 node decision tree?

Also accuracy is hard to talk about if you don't have the whole distribution you are estimating from; which generally we don't... Many highly accurate models (in the sense of working on the test set and n-fold xvalidation) underperform in production.