DB: So, control theory is model-based and concerned with worst case. Machine learning is data based and concerned with average case. Is there a middle ground?

BR: I think there is! And I think there's an exciting opportunity here to understand how to combine robust control and reinforcement learning. Being able to build systems from data alone simplifies the engineering process, and has had several recent promising results. Guaranteeing that these systems won't behave catastrophically will enable us to actually deploy machine learning systems in a variety of applications with major impacts on our lives. It might enable safe autonomous vehicles that can navigate complex terrains. Or could assist us in diagnostics and treatments in health care. There are a lot of exciting possibilities, and that's why I'm excited about how to find a bridge between these two viewpoints.

https://www.oreilly.com/ideas/machine-learning-in-the-wild

It doesn’t have to be all or nothing.

For instance, the network might only be used to decide among a series of actions, and those actions can still have limits (such as “car cannot travel faster than X” or “Y cannot change more than 3 times per minute”, or whatever). There is still an abundance of attention put into safety, as usual for the auto industry. It isn’t just a brain hooked up to an engine that is allowed to run rampant.

Depending on the context, 'turn the wheel left two degrees' can be a non-rampant, a very rampant, or a rampage-compensating action.

"Keep the car in a straight line" or "turn right, 10°" are the kinds of operation that will always be more performant, take less engineering, and more reliable to create as an old-fashioned simple math function than by using machine learning.

One would expect the output of the neural nets to be encoded in those.

It's the exceptional cases that get you:

1. "Keep the car in a straight line" => The road is closed or there is a detour, or the lane markers aren't visible (or incorrectly marked like on the Palo Alto stretch of the 101 where there are two sets of lane markers; one new (the "actual" markers) and another old but barely faded and still very visible.

2. "Turn right, 10 degree" => With what steering radius? What if the road is bumpy or sloped and the driver has to compensate with the wheel?

etc.

These corner cases are everyday occurrences and lend a necessity for something far more sophisticated than a mere ad hoc model to represent and control human-level driving behaviors with a finite set of rudimentary actions.

Actuator inputs like steering and pedals are almost always represented using continuous values rather than discrete sets for these very reasons.

As do humans (see the various cognitive biases). The question is: does the AI's black box result in a safer outcome?

And the answer is an obvious no, not even close, with the current state-of-the-art in deep nets. I could forgive an opaque model if it actually performed well.

Fun thought experiment.

An alien race landed on earth and demands to play a game of Go. We only get to play one game with them. If they win, our planet is destroyed.

Who would you trust to play for the human race if this scenario happened tomorrow? Lee Sedol or AlphaGo? Remember that we do not completely understand how AlphaGo reasons, it is still a black box to us.

Lee Sedol is also a black box, no?

Also, one shot for the survival of humanity is very different from billions of iterations of driving situations every day for the foreseeable future. A complete understanding of the system is much more valuable when you have the opportunity to iterate.

I'm not necessarily arguing for one approach; just saying that the analogy doesn't really apply to the case at hand.

> Lee Sedol is also a black box, no?

Last time I checked we can talk to Lee Seedol and ask him to explain things. We can ask him questions. We can have an intelligent conversation with him.

Human explanations for their decisions are often rationalizations after the fact. The explanations don't necessarily accurately represent how the decisions were actually made. Most decisions are made subconsciously based on intuition and emotion. So that intelligent conversion might not have any real significance.

Ya, that was my thinking. And likewise with the sort of hybrid, constrained neural net setup we're discussing here, you could likewise 'discuss' the constraints, inputs, perhaps even the thought process to some extent. But like a human, it couldn't tell you the exact causal path taken to arrive at the decision.

We often don't even have appropriate language for many decision processes. See: research into those that do vs. don't have internal monologues (virtual voice, basically) when reading and thinking, and associations to creative thought.

N.B: When I was younger I had no idea that others did those and thought they were fucking with me when they were describing this.

My general point applies to human thinking in general not just Go.

Example: Would you fly on a plane designed ultimately by a human vs an impenetrable black box?

Also there is a spectrum. Let us not pretend otherwise.

1. One end: No explanations.

2. Middle: Seometimes false explanataions and sometimes true explanations.

3. Other end: Always true explanations.

Are we really saying the middle is completely useless?

Yes I would fly in any large airplane which is properly certified for scheduled commercial airline service, regardless of how it was designed. The FAA has earned our trust and has a good safety record so if they tell me the design is satisfactory then I would believe them. I also wouldn't take the risk of flying in any non-certified experimental aircraft, again regardless of who or what designed it.

We have no way to determine whether an explanation is true, false, or simply a post-hoc rationalization. We like to believe that we can, but we're just fooling ourselves.

> We have no way to determine whether an explanation is true, false, or simply a post-hoc rationalization

If we have no way of determining whether something is true or false, then I can say the same thing about your own statement quoted above. I can just say it is false and go on with my life.

I sincerely hope you realize the obvious self contradiction :D

Logic 101. :)

There's no self contradiction. I never claimed that we have no way of determining whether something is true or false. I only claimed that we have no way of determining whether the explanation a person gives for how he made a decision actually matches his real mental process or motives. We can't yet install a debugger with break points and variable watches in the human mind; it's very much a black box.

Logic 201. :-)

Logic 101. :)

> I never claimed that we have no way of determining whether something is true or false

Everything can be cast as a declarative statement.

Matches(givenStatement, actualIntention).

The Black Box that can invent stories to give the illusion it understands it sub-conscious processes? How comforting.

I remember an article here on HN about somebody that trained a neural network to explain the decisions of another NN. I think it's fitting :)

Do we really need to understand the whole stack that goes into a decission?

That means we have to start with physics.

Yea the analogy doesn't hold. I was simply probing at the author's criticism (or fear) of deep learning as a flawed approach given the resulting model is pretty opaque.

To be honest, I would have felt the same way two years ago. Seeing all these recent advances in the field has slowly changed my mentality.

If emotions weren't a problem? Lee Sedol.

If adaptability is the metric, machines still didn't beat humans.

AlphaGo was trained on some set of movements (as were the other AIs that helped train it). It is not a pure game theory construction, so there is no reason to expect it to be as good against an alien as it is against a person.

AlphaGo for the simple reason that, in that scenario any human would be absolutely bricking and would be pretty much bound to make an unforced error as a result.

Definitely, quite scary, keeps me up at night. At least I know EXACTLY what my taxi driver is thinking....

/s