Why do you trust moody, inconsistent people with their own biases as your leaders in life-or-death situations then?

We have centuries/millennia/choose-a-time-frame of experience with what they do, how they tend to perform, and while it's not necessarily perfect, we know how to engineer around their limitations, again, with centuries/millennia/etc. of experience in that field.

Computer models are much younger, and we know they tend to have weird pathological corners, but unlike humans and their weird pathological corners, we have a much less firm grasp on what they are.

In many cases, humans have skin in the game, too. No computer model is yet sophisticated enough to be able to say that about.

There is also some irrationality in having someone to blame, etc. Certainly. But it's not the only part of the story.

Your comment made me think of something, perhaps in the larger assessment of the soundness/comfort-level we will/do have with AI decision making:

With the moody, inconsistent people, we can understand and evaluate their incentive reasoning - positive or negative and empathize with that understanding to assign a level of trust.

With any AI - especially to ones who didnt design/create it - any and all of its motivations, incentives are completely (emotionally) opaque to the recipient of the outcomes of its behavior.

I know we are saying the same thing - but unless a "receipt" can be given for a decision to why an AI / NN made a decision, people will not learn to build an understanding of a trust.

Otherwise, AI will always be hated by the humans who feel slighted by the cold decisions of a machine against them.

And that is clearly going to be the real future.

There will be anti-AI terrorism/activism and violence against systems, companies and countries that make life-bending decisions by AI against groups of people.

The seed to this future is the gatekeeping to capital/resources through FICO-like systems.

> There will be anti-AI terrorism/activism and violence

There's already been bomb attacks in Mexico against nanotechnology researchers, so it's not a big leap from there to AI/ML and other similar research (especially into GAI).

Absolutely. We all daily get practice in understanding humans, so I think humans are much better prepared to handle a rogue employee than a rogue model.

I'm not saying all AI is bad, but in the case of NN, they're full of biases known and unknown.

> they're full of biases known and unknown

Are we talking about AI or people?

Depending on where the training data comes from, both ;)

Like 'jerf says, we have many centuries of experience in dealing with people and people-based systems. We know the biases, the cognitive problems, the buttons to press and the buttons to absolutely avoid pressing.

More importantly, humans share the same brain architecture and work roughly the same way - you and me included. This makes it easier for us to understand other people (also having a part of the brain trying to directly model other people's minds helps). The way those NN-based black boxes work is completely alien to us (and each being a special snowflake made of hacks does not help).

People can be fined or jailed

That's an important point.

One of the depressing features of the 2008 financial crisis was the great number of industry titans who went smoothly from "My awesome insight is worth tens or hundreds of millions annually" to "Who could possibly have known?"

I would hate to see the same thing happen again when algorithms fail to work as hoped. If people build and deploy a system, they should be socially and legally responsible for its effects in the world. And executives should be held responsible by the rest of us. I'd like us to be done with the shrug-emoji-and-cash-the-bonus-check school of corporate governance.

And software can be deleted.

>And software can be deleted.

The software doesn't care and isn't intimidated by that fact... yet.

> Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained?

Almost all cars and aircraft today are operated by neural nets, the actions of which can't be explained.

As anyone driving a car can assure you, the actions can be explained, easily.

But are the explanations true? Are they deterministic?

If someone took ten minutes of video of you driving in city traffic; could you consistently explain all of your decisions that would be recorded?

It's very easy to tell yourself that your actions are consistent and explainable ( rational ) but if examined closely they aren't always consistent with any one set of heuristics. Following distance is one example; can you explain the decision making process by which you decide how much space to give the vehicle in front of you? Why it varies based on road conditions; weather or your estimate of your reaction times?

I'd also be worried about the model being biased. For example, perhaps the model is "better" because it uses correlated features it shouldnt be using (due to fairness laws, etc.)

There's a huge difference between trading algorithms and safety-critical control systems. If you're speculating with other people's money it's not entirely unreasonable to risk it using unexplainable models, provided you disclose your general approach to investors.

I don’t understand what you are asking for. 100% certainty before making any decision? Any decision in a messy real world that’s far more complicated than a simple math statement that can be proved?

"Would you want your car or an aircraft you're on piloted by neural net the actions of which can't be explained?" Actually, if the net is properly trained to reduce overfitting and is shown to work on out of sample data - yes. On the flip side of this argument, a person can do something inexplicable and invent a plausible explanation after the fact that seems and feels safe and correct, but is actually wrong.

Define: "properly trained".

With humans, we generally know the bounds for unexpected behavior. We understand tiredness, confusion, fear, distraction, suicidal thoughts and other factors. We also know how to screen people to minimize those bounds.

With ML stacks, we have no good grasp on bounds. They usually work, for some definition of working, up until they don't - and when they fail, it's in some absurd (therefore hard to predict) way.

humans also fail for all sorts of reasons that are unpredictable. if, for example, Waymo, demonstrates properly calculated lower accident rates with their cars than with humans, then at some point you have to agree with what the data says. We aren't there yet, but that doesn't mean we can't get there

If they demonstrate, sure.

But for that to be useful they also need to demonstrate that their model is robust under modifications - otherwise every time they retrain their NNs they should throw away prior safety records and start counting from zero. Because at this point, what would be the argument to keep it? With humans we know - from thousands of years of experience - that people generally don't go crazy when taught new things. With NNs, we know they're very sensitive to training data.

Well, you are making pretty general statements. "With humans we know - from thousands of years of experience - that people generally don't go crazy when taught new things"

they do crazy things all the time. Drink and drive, hijack planes and run them into mountains ( germany, malaysia recently ), text and drive, etc. etc. This discussion needs to be based on comparing actual data. Let's get there and see

This is one topic when you can sort of get away with general statements, because it's human interop - we all know how it works :).

I covered drunk driving and suicides in my original comment. This happens, but we know it does, we know how often and why it does, and know how to work around it.

What I was thinking about wrt. learning failure modes is this: when you put a person through 30+ hours driving course, they don't suddenly lose the ability to recognize trees or faces. The same cannot be said about retraining existing neural networks.

I'd love for the actual data to appear for analysis. Right now I'm worried about the very concept of using a black-box bag of statistical tricks DNNs are for safety-critical operations. How is it that we can't handle the problem of self-driving with more direct, stable and auditable methods?

I think you are putting a bit too much faith in human cognition: "when you put a person through 30+ hours driving course, they don't suddenly lose the ability to recognize trees or faces". They do. they get tired and their reaction time slows down. they get drunk, they get distracted. Humans do exactly this - unpredictably lose the ability to recognize and react in a timely manner.

I understand the point you're trying to get across, but to be clear about this tangent - it's not that I'm overoptimistic about human cognition. I'm aware how ill it is suited for the task of driving. I just put much, much less faith in DNNs.