I think it's also important to remember a lot of human models were abandoned because of the inherent bias and black-boxed nature of them (feelings, subjectivity, etc.).

This is one of the reasons the scientific method blossomed: objectivity, rigor, transparency, reproducibility, etc. Black boxes can lead to bad decisions because it's difficult to question the process leading to decisions or highlight flaws in conclusions. When a model is developed with more rigor, it can be openly critiqued.

Instead, we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.

To be clear, I'm not saying ANN/DNN models are bad, just that we need to understand what we're getting into when we use them and recognize effects that unknown error bounds may cause.

If, when the model fails to correctly classify a new data point, the result is your photo editing tool can't remove a background person properly... then so be it, no harm no foul. If the result is that the algorithm classified a face incorrectly with high certainty and lands someone in prison as a result (we're not there, yet) then we need to understand our method has potential unknown flaws and we should proceed carefully before jumping to conclusions.

> I think it's also important to remember a lot of human models were abandoned because of the inherent bias

The industry needs to stop misusing the term bias this way. Virtually every attempt to find this supposed human bias has failed. Latest public example was Amazon and hiring[1]

Bias is the tendency to consistently mis-classify towards a certain class or tendency to consistently to over or under-estimate.

Somehow the term has been hijacked to mean 'discriminate on factors that are politically incorrect'. You can have a super racist model that's bias free, and most models blinded to protected factors are in fact statistically biased.

It's not constructive to conflate actual bias with political incorrectness.

Operational decision making, whether AI or human or statistical, faces an inherent trilemma: it's impossible to simultaneously treat everyone the same way, to have a useful model, and to have uniformly distributed 'bias'-free outcomes. At best a model can strive to achieve two of these factors.

See: https://www.youtube.com/watch?v=Zn7oWIhFffs

[1] https://www.reuters.com/article/us-amazon-com-jobs-automatio...

Hold up. Bias is being used in two different manners because it has two differed meanings. When you are using bias in industry you are talking about a minor mathematical factor added to a learning rate. When we talk about human bias, we aren't. The term was never hijacked, it just has multiple meanings.

I’m not sure if he was using any of those known definitions of bias though. I’m not sure if I would define bias as the consistent act of mis-classification

Yes, mathematicians mean something different and specific when using the word bias. The average non-expert is not misusing the word. They are using the word to express a different -- and far more popular -- meaning.

Neither is wrong, but insisting that a naming clash carries any substantive significance on an underlying issue is just silly. Similarly, insisting that nonmathematicians should stop using a certain word unless they use it how mathematicians use it is a tad ridiculous.

Of anything, it's more reasonable for mathematicians to change their language. After all, their intended meaning is far less commonly understood.

If an algorithm is clearly sorting on irrelevant criterion, especially a black box algorithm, we normally assume it's a bug. It's not reasonable to reverse that, assume the code is incapable of being mistaken and say that obviously irrelevant criterion are somehow correct in an unknown way.

Amazon's problem is a bug, they even describe it's nature. And given how flawed their recommendation algorithms are, it's especially unreasonable to assume this one is infallible.

So that linked Reuters does not show a failure to find bias, if anything it shows a design error.

You say "obviously irrelevant criterion"

Data says criterion is an eigenvalue and no matter how hard amazon tried to blind the solution to that eigenvalue, the ML system kept finding ways to infer it because it was that strongly correlated with the fitness function.

This is the difference between political newspeak '''bias''' and actual bias. Amazon scrapped the model despite it performing just fine and being bias-free, because it kept finding ways to discriminate on a protected attribute which is a PR nightmare in the age of political outrage cancel culture. It's fine to explicitly decide that some attributes should not be discriminated upon, but this comes with a cost either in terms of model utility or in terms of discrimination against other demographics. There's no way around this. In designing operational decision making systems, one must explicitly choose a victim demographic or not to implement the system at all. There's no everyone-wins scenario.

The harm of the newspeak version of '''bias''' is that it misleads people into thinking that making system inputs uniform somehow makes it bias-free when the opposite is typically true. Worse, it creates the impression that some kind of magical bias-free system can exist where everyone is treated fairly, even though we've formally demonstrated that to be false.

No amount of white-boxing or model transparency will get around this trilemma. The sooner the industry comes to grips with it and learns to explicitly wield it when required, the better.

>No amount of white-boxing or model transparency will get around this trilemma. The sooner the industry comes to grips with it and learns to explicitly wield it when required, the better.

Agreed. The optima of multiple criterion will essentially never intersect.

But for Amazon, there is no evidence the tool was accurately selecting the best candidates. They themselves never said it was. After all, altering word choices in a trivial way dramatically affects ranking. On the points you mention, why should we assume their data was relevant or their fitness function even doing what they thought? If they were naive enough, they could just be building something that predicts what the staff of an e-commerce monopoly in a parallel universe will look like.

The most likely story is that they failed at what they were doing. Part of that failure happened to be controversial and so got unwanted attention. I would guess there were quite a few incredible correlations the tool "discovered" that did not get to press.

At any rate, their recommendation engine is more important and has been worked on longer yet it is conspicuously flawed. When their recommendation tool inspires awe then maybe we could take their recruiting engine seriously enough to imagine it has found deep socio-psychological-genetic secrets.

Supervised learning algorithms assume that the input data are iid of the future. This is not valid in most of the real applications. The observation that we see men more than women in programming does not necessarily generalize to the future. That's why online learning provides an exploitation vs exploration mechanism to minimize the bias in the hindsight. In many applications, people just forgot about this simple strategy and blame the bias caused by supervised learning to the black box model.

Of course, black box AI itself is not the right solution. As more and more cross domain multitask settings emerge, open box AI will gradually take off. It is about compositional capability like functor and monad in functional language. Explanable or not is just a communication problem which is parallel to the ultimate intelligence problem. It is very possible that human intelligence is bounded.

That is the fascinating thing about consciousness.

How do we "know" we see red and not green or vice versa? How do we "know" we are feeling heat or coldness? We just do. We feel different things in different, yet recognizable ways.

And these "feelings" are multi-dimensional, feeling heat and feeling color are not on the same axis. Red and Green seems to be on different points of the same axis and hot and cold are points on some different axis, other dimension.

How many different dimensions of "feelings" are there? Note they are not the same as different senses. There can be many different feelings associated with things we see and recognize that way: colors, and shapes and lightness vs. darkness etc.

Physical sensation seems simple tbf. Any robot can sense temperature or color gradient.

It's the decision making that's complicated.

To make a decision on whether it would like chocolate ice cream or not (before trying it) a brain will take all the data gathered over time, cross reference it with related data (does chocolate taste good alone? \ did I like it? \ It looked rather inedible. \ was it high cocoa content or not? \ Chocolate yoghurt seems similar, but I did not like it. \ I do like cold, but only when it's hot outside. \ My friend says chocolate ice cream tastes good. \ My other friend says it tastes bad. \ trust or discard opinions? and much more).

All within minutes, seconds even.

An AGI would need an absolutely massive database of knowledge similar to the one acquired by a human over the first 20-30 years of their life, if it's to be truly general.

It would also need bias correction, and maybe 2-5 other AGIs to form a consensus before making any decision properly.

And even that would not be enough to fit in with humans - if you want an AGI to make the objectively right decisions, it will have to ignore many human feelings/biases.

If you want it to fit in with people, it will have to make "mistakes", i.e. ineffective decisions.

> Physical sensation seems simple tbf. Any robot can sense temperature or color gradient.

It can measure them and process it as a number, but when does sensing come into play? Does a red sheet of paper recorded by a simple camera evoke the same "feeling" of redness inside the camera that it evokes in a human being?

The flippant answer is "of course not, the camera has no consciousness/perception". The question is then how and when this feeling of "redness" gets created and what are the necessary conditions for it to happen.

By sensing, I mean just that, reading input. You mean reasoning/decision, I guess.

That would come from learned data, experience, imo. A child who's never touched a red hot electric stove, for example, would not have any bias towards it. No fear, no love, unless they previously interacted with one. They would try and get close, watch it, smell it, touch it, to learn more.

The most interesting part of consciousness is how does one decide based on incomplete information? If you need to learn more, where do you find the raw information? It seems to be done subconsciously, some people have a higher affinity for learning/fact-finding than others. But all people are pre-wired to learn from others, distributed computing works best heh.

I guess you're asking the same question as me, where is this "programming" and how is it created? It's not just raw experience, it seems to be genetic/evolutionary. A sort of basic firmware to bootstrap further learning.

I find it fascinating, it would seem the brain never stops processing the data it acquires. During the day, and during the night, it always runs learning jobs.

I agree with your thinking on learning, so I have nothing to add, but it doesn't sound like we were referring to the same thing. I'll try rephrasing, though I find this subject extremely hard to communicate effectively.

When you're looking at a red sheet of paper, reading the input in the form of a signal is not the only thing happening. The signal is definitely read and passed along to other brain circuitry for processing, but somehow another thing happens: the experience of red (which is what the other poster called "feeling"). This experience of red is fundamentally different from the experience of green or blue or from the complete absence of looking. This experience is what I was referring to.

This experience seems to happen without any prior knowledge or exposure to the color red. The first time you stumble upon red light, this experience of red arises. You can tell red from green by the marked difference in experiences, but you cannot explain this difference in words to someone who hasn't experienced red or green themselves.

How does this experience get created? Does any sensor (such as a camera) have such experiences? Why not?

I don’t know if you’re asking rhetorically or not, but the honest answer is we don’t know yet. Those sensory experiences are called “qualia” in philosophy and cognitive science and they are not well understood yet. You might enjoy the paper “What is it like to be a bat?” https://warwick.ac.uk/fac/cross_fac/iatl/study/ugmodules/hum...

Yes, it was rhetorical in an attempt to get the point across. I try to refrain from naming qualia when first explaining it. I've read Nagel's paper, but I haven't thought about it for some time so thanks for reminding me.

I usually see the Turing Test as a thought experiment for showing the irrelevance of qualia for AI, among other philosophical objections (e.g. Chinese Room).

These feelings you're describing are called qualia (https://en.wikipedia.org/wiki/Qualia), in case you're not familiar with the term.

> When a model is developed with more rigor, it can be openly critiqued.

Yes. And the models developed like this haven't solved the problems the current black box models have.

> we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.

There are ways being studied to check for sensitivity to parameters, biases, etc. But in the end, reality is difficult and there's no way of dealing with that, or "getting the right answer" every time

Sure, human pattern recognition is poorly understood, but it's a system that evolved over millions of years. We should expect more from something we ourselves designed.

The question of why we would choose a black box model over something interpretable that works equally or nearly as well still needs to be asked, especially if there is a misconception of black box models working better than interpretable models due to current trends and the hype cycle.

> held legally accountable

I think this misses the point. It is not about accountability but about getting the right results. The right question is where we need an interpretable model to eliminate biases or otherwise satisfy ethical requirements. Often you can't know whether a decision was ethical without knowing how it was reached. Would you accept a trial verdict given by AI? What about a lending decision, or a job promotion? If you need to justify a decision and not merely reach a decision, you need something interpretable.

>human pattern recognition is ,,, a system that evolved over millions of years. We should expect more from something we ourselves designed.

Why? Human intelligence evolved over millions of years. And we're struggling to keep our changes to the world from killing us.

Because we design it, so we have the ability to choose its characteristics. The current prioritization of 1% improvements in artificial benchmarks over all else is a disease that is actively retarding progress in the entire field.

Political systems are also designed. If human behavior is in danger of killing us, we must look to the political systems we have created and solve the problems there.

>how do we ensure machine learning is trained and verified in ways that don't encode bias

Or, more commonly, people work to ensure their machine learning is trained and limited to not specifically notice any patterns of fact that would be politically unacceptable to notice and make use of, even if it was rational and useful for the task at hand.

The status quo moral system of our society rests on a collection of objectively false beliefs about the physical world. Most humans know enough to avoid noticing these facts because of the social cost and possible societal negative impacts of shaking those moral foundations. It's not even necessarily wrong, since agreed-upon lies can do a lot to reduce conflict and cruelty. But machines need to be specifically trained to not notice these facts. You need to teach the machine not to notice the emperor's nudity.

And what's even better is that this'll never get acknowledged in the literature. It's one of those self-hiding facts which exists if you're willing to notice it yourself, but which no authority will ever tell you (I love these, wish there was a word for them).

If we were able to provide perfect datasets you would be correct. As it is our datasets are imperfect and ML models are only as good as the training data.

> I think it's important to note that human pattern recognition is basically black-box as well.

Mammalian pattern recognition is fairly well understood. We have a good understanding of the information path from the eye to V1 in most model animals. So sure are we of this pathway that we've successfully implanted ships into blind patients and restored sight (results vary a LOT, though).

I know this may not fit pattern recognition in a super-philosophical sense, but damned if it's not a ways down that road. We're literally shocking a person's brain so that they 'see' again (!!!). How V1 feeds into the rest of the brain is VERY active research and has had a lot of successes, but we're a long ways from a 'black box' these days.

Honestly, the auditory pathways are a lot better understood, as they seem to be 'older' and specific tones are represented physically in the mammalian brain (tonotopy). As such we know a lot better (relative to sight, smell, taste, etc) how tones are encoded and how they elicit specific recognitions and responses (this work is NOT straightforward).

In Neuroscience/Biochem, we are getting a LOT closer, but yes, we are not there yet by any means. The 'black box' idea of the mammalian brain is not going to last another 500 years, likely. We're marching along with good time as a species.

Knowing how to input data to the visual cortex doesn't mean that we know how it works. It is the same with neural networks, we know how to pass data to them and how they do the math, but we can't trace back its reasoning to know why it comes to the conclusions it does.

We have a very very good idea of the mammalian eye-V1 pathway. It's not a straightforward path and describing it in a HN comment is not going to be useful to anyone. Explaining the complex, but well known, pathway is a job best done by other people than myself. We are well aware of what 'data' processing the mammalian brain does at nearly every synapse, and it is a LOT. Right form the cones and rods, data is being highly processed (synapses are VERY noisy, as it turns out). So while more research always comes up with more questions, we are fairly confident about the eye-V1 pathway (contrast that to other organs, like the vestibules, where we're fairly in the dark still).

That said, wait, what? We can't do the back-propagation calculus in a NN? When has that been true and for how long? I thought it was fairly straightforward to know the weights of the connections of the nodes in your network. It's just a tensor you grep for, right?

Black boxness isn't a property of the thing, it's a property of our understanding of the thing. Just because we don't understand the how pattern recognition in humans works, doesn't mean the process has no internal structure, or that we couldn't potentially understand it in the future.

Right, and currently we understand neither human vision, nor CNN vision - they are both black boxes. I don't think anyone said they must remain black boxes forever.

Edit: I guess you were referring to this:

> Robust pattern recognition based on 100+'s of factors is just inherently black box.

I think he meant " ... given our current knowledge."

Yes, that’s true. But human decisions have accountability - algorithms don’t have accountability. If entities who deploy algorithms would be legally required to take accountability for algorithmic decisions, there would be no issue with black box algorithms.

I agree in spirit, but a lot of human behaviors, derived from decisions, don’t have consequences. Did Nestle face consequences for their decisions to promote formula, etc?

Let's not confuse accountability of decision making entity with consequences faced by the accountability. I was referring to the former, not the latter. The former is a case for law, the latter is a subject of law enforcement. In the case of Nestlé, you have a clear assigned accountable entity. We don't have that with algorithmic decision making (yet) where organizations wash off their hands saying "Oops! It was an algorithm, we didn't do anything!" Weak law enforcement is not a reason to not have proper laws in place.

Why would organizations not be responsible for decisions made by an algorithm? After all they are still the entity who decided to use that algorithm, and to execute the decision.

Sure, they say "Oops! It was an algorithm, we didn't do anything!", but is that any different than management saying "Oops! It was just a rogue employee, we didn't do anything!" For anything sufficiently consequential or systematic the second excuse doesn't work, so why should the first excuse work?

> We don't have that with algorithmic decision making (yet) where organizations wash off their hands saying "Oops! It was an algorithm, we didn't do anything!"

That's what individuals within companies do all the time.

You're onto something, form a limited liability company for your AIs and hire them as contractors. There, a clear assigned accountable entity :)

> I think it's important to note that human pattern recognition is basically black-box as well.

Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

> But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion.

No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

> So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

You can already do that. Just change that number in the input and see how the output changes. To that extent, even the most black box AI model is more transparent than human decision making.

> You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

Well, I guess it depends on how self-aware a person is. I think the biggest danger is trying to rationally explain your decision when in fact it was based mostly on your feelings, in which case I agree that the explanation is "basically a lie". One needs to be honest when something is not based on a fact but on a feeling to prevent pointless discussions. (If I hold an opinion based on a feeling then you cannot convince me that I am wrong by giving me facts.)

> You can already do that. Just change that number in the input and see how the output changes.

Makes sense. But I guess transparent models would still be generally preferable because you can fully understand how the output is produced, whereas in black-box models you might have to ask quite a lot of questions to get a feeling for it, but even then you can't be sure that you have a full understanding of it.

> even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

You punctuate the second sentence as though it were of secondary importance. But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers. I put it to you that having questions answered with plausible fabrications, is actually a significantly worse situation than not yet being able to ask the questions at all. At least in the latter situation, we know what we need to be working on.

> But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers.

Hmm, too me it feels like I can explain the reasons why I came to a conclusion in many (but certainly not all) cases. You "just" need to clearly identify your feelings and emotions and separate them from your rational arguments.

Anyway, these are our own shortcomings and of course don't have to be adopted by any artificially built black-box model.

This is actually false. Early psychologists thought that humans could know everything about their own brain's processes, but observational psychology proved most of their assumptions wrong. It's called introspection, and it's almost always not indicative of a person's actual inner workings.

Expert systems from the 70s and 80s had a capability similar to this. They could explain how they reached a conclusion by reporting the rules they used to get there. The problem was that interviewing experts and coming up with a huge rules database was a ton of work and didn't scale very well.

Maybe the next direction in AI will be to bridge the gap between expert systems and black box models?

> we can ask them questions in order to find out how they came to a particular conclusion.

For the most basic cognitive tasks, we typically can't.

If you show someone a picture of a cat and a dog, they can easily recognize which is which.

If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...

I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.

Introspectable Higher level reasoning is irrelevant. Any introspectable higher level reasoning a person can do has already been automated for efficiency.

Wait, what? Where has this been automated?

Humans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.

But that explanation is likely full of cognitive bias, and all sorts of subjective experience.

Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.

Object recognition is firmly in the realm of Neuroscience. We've had a good idea what the eye-V1 path is doing in mammals for a few decades now. To the point where we are implanting chips into the brain to (crudely) re-establish vision in blind patients. Though the research is very much on-going, it is being better understood with each passing year.

Respectfully, unless I'm not understanding what you mean by this, that seems very far from object recognition. You're replicating the data stream that would normally be coming from the optic nerve/retina, but that's not the same thing as understanding how the data of an object's image relates recognition of that object. If I've never seen the entire Mona Lisa before (but maybe have seen portions of it), your proposed system could force me to see the painting but it can't force me to recognize it as something I have seen, nor force me to know that it is a famous painting by Leonardo da Vinci currently on display at the Louvre.

I don't think you are understanding. But Bio isn't meant to be easy.

At each synapse a significant amount of data processing occurs, right from the rod and cone cells in the retina and at nearly each synapse along the way. 'Data' is more-ish conserved into V1 where all hell seems to break loose, path-wise. After V1 and in nearly the entire cortex, these data are distributed in unique and very complicated ways throughout the brain. Most of the mammalian eye-V1 pathway is well understood, but that process is too much for a comment on HN. But, I want to stress, we have a very very good idea of what is going on.

That said, the research seems to indicate (highly debated still) that there are specific synapses that encode 'the left side of grandma's nose in dim lighting from 10 feet away' and other many other things. Again, I cannot stress enough that such 'grandma' synapses are not firm science and there is a LOT of research still ongoing. But the evidence seems to be pointing that way at this time. Such cells are fed this information from V1 and, likely, a lot of other places. When such 'grandma' cells fire, they then send that signal out to other cells they may be connected to. Such a system is likely replicated many times all over the brain; there are many copies of 'grandma' cells and they are wired up in many different ways. The physical location of 'grandma' cells in highly unique, if they exist at all. Again, research is very much on-going.

One more benefit: an interpretable model could at least give some hint when the model is not working as intended (e.g. debugging).

Sometimes I have the feeling that this is exactly why so many people prefer black-box modeling. You are not going to be criticised and you don't have to do all the hard work of software development.

It's true that humans are black boxes but there are whitebox techniques such as the scientific method and algorithms that are justified step by step.

I think the article tries to compare black box learned algorithms to explicit algorithms. Humans being a blackbox is more of a philosophical issue from that perspective.

> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Isn’t the first sentence just an example of the second?

It's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions.

That's not the same question as whether there were feelings involved, or whether the process was conscious.

"The point is that the explanation given does not correspond to the process through which the decision was truly reached."

Sure, but that doesn't mean "a weighted combination of feelings" is an accurate description of the true process. That's tendentious and speculative. What I think we can be sure of is that there are parts of our decision making that we are not consciously aware of, but precisely because of that, one should be cautious of assuming how it works even if you're not doing it by reflex. Psychological explanations are the rationalizations that people make after the fact.

Parent wasn't saying that the weighting process was rational. The weights are also feelings.

As long as we don't know what the weighting process is, we shouldn't make unfounded statements about how it works. You're asserting that it exists, but nobody knows anything about what it is. It's a program that you haven't read the source code for.

At this point we seem to be reasoning by analogy. We'd do better to just study the fact of the matter. Psychologists do not tend to discuss this topic in terms of weightings.

https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...

http://www.theunconsciousconsumer.com/consumer-psychology/20... (regrettably thin on citations)

https://academic.oup.com/scan/article/6/4/460/1648209

https://en.wikipedia.org/wiki/Rationalization_(psychology)

I'm not reasoning by analogy; I'm disputing that it's useful.

If you look at the physical world with objects that have relationships and characteristics, and describe a mental world that also has discrete entities that interact in designated ways, then I think you are reasoning by (unfounded) analogy.

> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Does it matter? If a brain counstructs post-hoc argument for given decision and the argument is defensible against other arguments, then the argument itself is good enough substantiation of the decision, even if it is different from the actual decision process.

> because despite being less accurate, they can have designers who can be held legally accountable and liable

Are you implying you cannot be held accountable for black box models?

An anecdote, but this is a criticism being aimed at YouTube currently for it's demonitisation scheme.

The argument being made is that even though the demonitisation of certain classifications of content is immoral (e.g. demonitising any and all LGBT content, even when family friendly), it is tacitly being allowed to happen due to YouTube's stance of "the algorithm" making the decision, not them.

Which may lead then to have a position of "supporting" LGBT as a company while simultaneously being able to demonitise said content, which could hurt their business in actively hostile to LGBT countries.

This isn't proven, but it's an example of how it could be used to absolve yourself of blame and point it at an inscrutable system.

I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?

I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain.

AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for using the system. You can still hold manufacturers and businesses accountable for promising a degree of accuracy that their system couldn't meet.

In the same way, if my dog bites someone, and I get sued and argue, "animal motivation is really hard, we don't know why the dog chose to do that", I'm not going to win the case. The reason it happened is because I didn't leash my dog.

The reason the black box hurt someone is because an accountable human improperly used it or inaccurately advertised it.

This is tort law, the worst defined laws we have. Liability is far more complex that and usually governed by irrational humans or arbitrary bureaucratic rules contrived by powerful people protecting their interests.

But how do black boxes change that? It's still the same people operating under the same laws.

A few commenters here posit that without a black box, designers might be held accountable. If there's a system where software manufacturers would be held accountable for bugs, is any judge going to say, "well, the software is bugged, but they don't know how to fix the bugs, so they're off the hook"?

If there's proof that Twizzlers are poisonous and kill people, the Twizzlers manufacturer can't say, "but we don't know why, we just threw a bunch of chemicals in a vat at random without writing down the labels. So therefore, it's not our fault."

I don't know, can they? I'm not a lawyer, maybe everything in our legal system is way more horrifyingly broken than I assume.

There are payment systems based on human facial recognition. If the model failed, it means losing money. Apparently you can not deny liability because of using a black-box model.

Not all human thinking is the same. How you decide the amount to tip your waiter is not at all the same as how your doctor diagnoses you. And at we have a zillion years of evolution towards understanding and judging and conversing with each other's black boxes.

This is an incredibly salient point. Others have also pointed out that there are numerous applications in which black box models appear to offer significantly greater accuracy than interpretable models, bolstering the notion that this article is a bit overstated.

However, in this article and elsewhere Professor Rudin has cited compelling evidence of cases in which black box models have been demonstrated to be no more accurate than interpretable alternatives. I feel this fairly justifies the question in the title of the article. For example, based upon available evidence, it appears reasonable that some onus should lie on the creators and buyers of COMPAS (a proprietary black box recidivism model) to demonstrate COMPAS actually is more accurate than an interpretable baseline. While it may not be the case, as the article seems to suggest, that in all modeling cases there is an interpretable alternative with comparable accuracy, in cases which there is, there doesn't seem to be any justification for using a black box model.

On the matter of "human-style" interpretability, we are brought to the difference between "interpretability" and "explainability." Humans have a complex capacity for constructing explanations for the thoughts and actions of ourselves and others (among other things). As OP points out, a lot of famous psychological experiments by Kahneman and others have shown how much of our reasoning appears to be post-hoc, often biased, and often inaccurate (in other words, human explanations are not actually true transparent interpretations of our thoughts and actions). However, we humans do have a powerful capacity to evaluate and challenge the explanations presented by others, and we are able to reject bad explanations. For those interested, a great book on this topic is "The Enigma of Reason" by Mercier and Sperber (https://www.hup.harvard.edu/catalog.php?isbn=9780674237827), but the gist here is that we must understand that while explanations are not the same as transparent interpretability, they are still useful.

I would conjecture that at some level of complexity (which some predictive tasks like pixel-to-label image recognition seem to exhibit), true end-to-end interpretability is not possible -- the best we can do is to construct an explanation. However, two very important points should be observed when considering this conjecture: 1. (Professor Rudin's point in the article) In cases which are not too complex for interpretable models to achieve comparable accuracy to black-box models, we can and should use them, as they offer super-human transparency at no cost in accuracy. 2. Constructing no explanations (or bad explanations) is not the same as reaching the same level of semi-transparency that humans offer. If we want to use human interpretability as a benchmark, black box models with no explanations are not up to par.

Was the fact that objects fall down a "black box" before Newton came up with some equations?

I don't think its fair to describe CNN's and human vision as being black boxes in the same way.

The fact that objects fall down was well understood, obviously. What wasn't understood was the mechanism that makes this happen. Why did that apple fall down instead of up? Where exactly is down a round earth? Where is down when I stand next to the Himalaya? Why don't planets fall down?

In my eyes this is very similar to our lack of understanding about the inner workings of a CNN, or human vision. We understand the input and the output and their rough relationship, we just don't understand how the move from input to output happens.

I see it differently.

The distinction I would make is that even before Newton, people could make useful predictions about the trajectory of falling objects. My dog does it well, every time he catches a ball. He even seems to grok the effect of wind on a frisbee.

We have no way of predicting how a CNN will react to an input. GANs really illustrate this point. I think this is an important distinction.

> And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?

Decision trees are black box models and are highly accurate. (Think XGBoost, LightGBM, Random Forest, etc)

Not sure what point you are trying to make here.

Those models are ensembles of decision trees. A single tree is interpretable, but sensitive to overfitting.