News for hackers. Stuff that matters.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by gaius on Sunday May 07, 2017 @05:27AM (#14284609)
requires all algorithms to "explain" their reasoning

It's about the application. If you are detecting cat images on a social network, then a few false positives or negatives are totally no big deal. If your algo is making investment decisions or is being used as evidence in court then it is entirely reasonable to expect it to have a rock solid causal chain easily comprehensible by a layperson. The example in the article is noise reduction and the method was "we tweaked it until we got the results we wanted". In many fields that's called "overfitting" (or "fitting up") and is a big red flag.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by backpropaganda on Sunday May 07, 2017 @05:38AM (#14284642)
All machine learning models are capable of overfitting. Deep learning does not have a special relationship to overfitting, and in fact, there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1].

Citing certain examples of misuse of deep learning by folks who "tweaked it till it worked" doesn't say anything about deep learning at large. The same team would do the same with a much less powerful model.

Of course, being able to explain is a very valuable trait. But there's just no evidence that deep learning methods are any less amenable to interpretation than SVMs or decision trees. The latter models' "interpretation" is mostly stuff like "feature 543 and 632 were on" , while deep learning methods can not only do that, but can also synthesize examples of characteristics that the model looked for [2].

[1]: https://arxiv.org/abs/1611.03530 [2]: https://arxiv.org/abs/1605.09304

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by gaius on Sunday May 07, 2017 @06:30AM (#14284771)
Deep learning does not have a special relationship to overfitting

Yes and no. Let's say you have a time series and you know there is some pattern in it and you are interested in predicting what it will be at time t. A Fourier analysis would give you a formula to plug numbers into that anyone can understand. NN or DL might give you as good, or accounting for noise, even a slightly better prediction when backtested but now you need to explain "why did it do this" - to a financial regulator who is looking at insider trades, or a commission looking into why a piece of equipment failed killing the user (say, a self-driving car), or a newspaper who's noticed more people of race X are being turned down for mortgages. That's why I say it is dependent on the application of the algo.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by YeGoblynQueenne on Sunday May 07, 2017 @11:22AM (#14285865)
>> [SVM and Decision Trees'] "interpretation" is mostly stuff like "feature 543 and 632 were on"

SVMs are well-known black-box optimisers of numerical parameters so not more or less interpretable than neural nets, but decision tree learners are a different matter. The model they build is symbolic- a propositional logic theorem that can be output in a format that is directly inspectable and interpretable by a human being with an understanding of propositional logic.

It's true that once decision trees grow beyond a certain size they're hard to interpret, but even so they can make a lot more sense than a heap of numbers in a large graph.

Neural nets used for image processing are special in that they can output their activations as images that are directly interpretable by humans, but that's about the only application where we can really tell what a deep network is learning.

Edit:

>> All machine learning models are capable of overfitting.

Aye, but deep nets by definition build extremely complex models with millions of parameters and so are especially prone to overfitting. Bias-vs-variance and all that.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by yorwba on Sunday May 07, 2017 @05:56AM (#14284682)
Then requiring algorithms to explain their reasoning should give neural networks an edge over other methods, if they are really easier to interpret.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by Asooka on Sunday May 07, 2017 @07:27AM (#14284927)
But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct?

Edit: I should add that I worked on a similar problem. I can't reveal all the details, but it was about scoring pairs of objects to say how much one object is like the other. At first we used an off-the-shelf system that only spit out abstract values, so you can say "A is like B with a value of 5, but it's like C with a value of 5.17, so A is more like C than like B". The problem was those scores didn't correspond to any explainable metrics, so we switched to a custom system that could spit out a normalized value (0 to 1) and tell you exactly which qualities were alike and how the score was formed. People were NEVER happy with the values. Never. Everyone "felt" that these two should be more alike, or there should always be at least one object that has at least 0.8 likeness to another, etc. Plus, the values were now non-transitive, i.e. A can be 0.9 like B, but B is 0.2 like A. After about a year of tweaking we switched back to a non-explainable system.

Of course, the "non-explainable" system is perfectly explainable, it's a computer program and it isn't even deep learning, it's just long equations, the explanation is just "this long equation results in this value". The semantics of that equation though are very hard to impossible to understand. Plus, I don't think we can have a semantic explanation of a deep learning system precisely because we use deep learning when we can't semantically specify an algorithmic solution to the problem.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by gaius on Sunday May 07, 2017 @08:00AM (#14285026)
But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct?

It depends who you are explaining it to, of course, different audiences will have different standards, and will have or have access to, different kinds of experts to validate the explanation.

If the question is "why did your self-driving car hit that pedestrian?" for example, the standard would be pretty high. "It works most of the time but we don't know why it failed this time and we don't know when it might fail again" isn't going to fly. If the question is "we've just busted this trader for insider dealing, and you made the same trades at the same time, why?" then "the computer just told me to for no particular reason" isn't going to go down well either. You would need to show how data that you legitimately had access to reproducibly leads to exactly the same decision being made.

What do you do if you don't like the explanation? That's easy. Someone's going to jail. Don't let that be you :-)

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by singham on Sunday May 07, 2017 @03:48PM (#14287231)
Consider the following case.

Let's say we ban development of self-driving cars since the algos can't be explained. Then we will never know the benefit we might reap from adopting them. We know humans make mistake. But with self-driving cars, the error rates in future might be very low as compared to human. We as a human society will never experience this future because we had this silly idealism that all algos must explain themselves in human terms.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by gaius on Monday May 08, 2017 @02:08AM (#14289588)
Let's say we ban development of self-driving cars since the algos can't be explained.

That's a false dilemma since it's by no means proven that the algos CAN'T be explained - just that we don't know how to extract the explanation from the model yet. It's entirely reasonable to suspend real-world use until the maths catches up.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by singham on Monday May 08, 2017 @01:49PM (#14293867)
But we would never reap the benefits in the meantime. That was my main point. And this becomes crucial if the saving of deaths from accident is substantial as compared to deaths that happen due to AI errors.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by Asooka on Sunday May 07, 2017 @12:16PM (#14286131)
> why did your self-driving car hit that pedestrian?

It didn't see him/her. Which is the extent of an explanation a human can give you. Try to picture wanting a person to explain why they didn't see a person coming from the side. We don't know what's happening inside our own heads.

That's my point - we're recreating an inherently non-explainable system in software, because we can't explain how it works. The algorithm is so complicated, we can't code it explicitly, so we let the machine figure out the semantic-free equations. All is not bad though, you could always dry-run the algorithm at home, i.e. ask your car "how visible am I with these clothes on", and just not walk outside with low-visibility and low-recognition clothes. Some countries already require pedestrians to wear reflective clothing in the evening to make themselves more visible. I suspect we'll have similar rules for self-driving vehicles: you must wear recognisable clothing when walking through traffic.

What I mean about "what if you don't like the explanation" is something else. Let's say you use a deep-learning software to determine sentences. The software spits out "5 years, because this is a black male". Do you accept that? It has been proven effective in many previous cases, can you accept that it has independently discovered a correlation between the person being black and the length of the sentence? Let's say you entirely remove the person's colour and sex from the dataset and re-train, then it will give you shorter sentences than you expect, because every judge previously had subconciously taken all these variables into account. Or it will discover a correlation between wearing e.g. adidas and alcoholism because Slavic people have a higher rate of alcoholism and wear adidas more often than others.

Which comes back to my core argument: If you know the acceptable metrics and algorithms, why are you using deep learning? If you don't know the acceptable algorithm, how can you verify the explanation?

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by gaius on Sunday May 07, 2017 @01:02PM (#14286327)
It didn't see him/her. Which is the extent of an explanation a human can give you. Try to picture wanting a person to explain why they didn't see a person coming from the side. We don't know what's happening inside our own heads.

There is a world of difference between "this individual driver made a human error" and "software that can't reliably see people is driving a million cars now". The developers of the software would need to demonstrate they understood why the software failed in this situation and how they are going to fix it.

I suspect we'll have similar rules for self-driving vehicles: you must wear recognisable clothing when walking through traffic

Bear in mind that we can't even make cyclists now wear hi-vis or fit lights - making the entire population wear certain things for the benefit of job-destroying technology isn't going to get many votes!

If you know the acceptable metrics and algorithms, why are you using deep learning?

I disagree, because it's a matter of scale and usefulness. An algorithm that detects pictures of cats can answer the question of "why do you think that's a cat?" with "because it looks like a cat" and that's a perfectly reasonable explanation that a human might give[0]. But the stakes are raised when you start to deal with real people in the real world. "Why did you turn down that loan application" or "why did you add that name to the no-fly list" or "why did you reject that job candidate" need a bit more exposition.

That's a self-correcting problem because no-one sane will risk their business on software that "inexplicably" just happens to reject everyone in a protected class because there happens to be some vague correlation between members of that class and some undesirable activity or outcome.

[0] Except http://www.bbc.co.uk/news/technology-33347866 of course

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by intended on Monday May 08, 2017 @07:21AM (#14290681)
> you must wear recognisable clothing when walking through traffic.

Which would make Driverless cars a non starter in the real world.

Dear God, I can't imagine the poor coder who has to dictate fashion so that car firms can ensure they don't hit people.

That addition reneges on the basic expectations people had of driver less cars in the first place, so now why would they have any motivation to let them happen?

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by xherberta on Sunday May 07, 2017 @10:02AM (#14285485)
As Cathy O'Neil points out, black-box algos are deciding things that have a huge impact on individuals' lives: which teachers get fired, which prisoners don't get parole, and which loan applicants get approved and denied. In the latter two cases, the algos are tougher on minorities.

It's troubling when we don't know why an algo is racist. We need ways of checking that these influential algos aren't reinforcing trends we would rather diminish.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by Houshalter on Sunday May 07, 2017 @01:04PM (#14286340)
No one cares about human judges that decide parole or to fire teachers. Humans are vastly worse. Attractive people are much more likely to get hired or get shorter prison sentences. Judges tend to give much harsher sentences just before lunch when they are hungry. Their predictions about what people are likely to reoffend are often worse than chance.

Additionally, the study about the "racist algorithms" was fraudulent. Their results were "almost statistically significant". I.e. not statistically significant. Compared to human judges which have well studied biases by race. There's nothing remotely fair about human judgement and you should always prefer an algorithm. In almost every domain they could find, researchers have found that even simple linear regression beats predictions of human experts.

And this has huge effects on our society. Humans making biased hiring decisions leads to mass discrimination against certain groups and very suboptimal employees. Having humans make loan decisions, means much higher interest rates, more people go bankrupt, and the economy grows much slower.

The EU banning the use of algorithms is just absurd.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by intended on Monday May 08, 2017 @07:15AM (#14290651)
Yes, but Humans can go to jail.

You assume that your opponent is reality.

your opponent is other human beings. It takes precious little for a motivated person to learn how to hide malfeasance behind a black box.

Faith in un-corrupted black boxes should be considered the same way as faith in the un-coruptible internet

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by Houshalter on Monday May 08, 2017 @01:42PM (#14293805)
No one has any incentive to corrupt the black box. There are many, many cases where systems have terrible incentives that ruin everything. But I don't see how this is one of them. E.g. the bank has every incentive to make it's loan system as accurate as possible.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by xherberta on Wednesday May 10, 2017 @12:13PM (#14309351)
This is a good point, and to her credit, Cathy O'Neil is careful to say that humans are worse. She's arguing for better algos, ones that are carefully thought out to avoid the worst of these pitfalls.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by p1esk on Sunday May 07, 2017 @02:51PM (#14286898)
I think he was complaining about the algos' decisions being hard to understand, not them being unfair. From your examples, it seems like we more or less can understand human decisions.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by gaius on Sunday May 07, 2017 @11:00AM (#14285761)
No-fly lists as well.

reinforcing trends we would rather diminish

Well, of course they are, at the end of the day all any predictive algo is doing is extrapolating a trend. It usually requires serious regulatory intervention - such as men being charged more for car insurance because "the computers said so" until legislation was passed barring sex as an input to the model.

Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by sdenton4 on Sunday May 07, 2017 @12:29PM (#14286186)
The black box nature of the regression can make things difficult, though. If you have a couple other parameters which indicate 'black male' with high probability, you might find layer one reconstructing the notion of gender and layer two making decisions based on it.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by moultano on Sunday May 07, 2017 @06:06PM (#14287965)
Much better to include the feature in the training, and exclude it from the inference. That actually cancels out the effect, rather than just incentivizing the model to reconstruct it.
Re:Deep Learning’s Impact on Image Processing, Mathematics, ... by tensor on Sunday May 07, 2017 @04:08PM (#14287326)
For your court example, if you have human oversight it doesn't matter. For example, machine finds evidence, human looks at it the found evidence, verifies and explains it, then no problem. Also, I'd like to point out that much "evidence" in court is very far from rock solid. I'm always curious why we hold machines to higher standards than humans.