I think a valid concern is that ML methods are being applied in critical, real-life scenarios without some practitioners being aware of flaws (bias, adversarial attacks, privacy issues) and without any theoretic safetynet that helps them reason about how these systems will behave. James Mickens discussed this recently in a keynote: https://www.usenix.org/conference/usenixsecurity18/presentat...

Maybe the only way to make steady progress here is to blaze ahead and rely on empirical evidence, whilst the theory is inevitably fleshed out. That's often the counter argument - that we do not know how the brain works, but rely on it nonetheless.

We don't rely on the brain though, at least not on any single one. Any system that relies on human brains alone without cross-checking or, ideally, much simpler automatic systems, will eventually malfunction terribly. A large organization never wants to rely on a single person's judgment for anything, a programmer wants automated systems checking their work, etc.

We rely on a single human to drive a car.

In medicine, machine learning systems will work alongside other brains.

What are the instances you’re imagining where a group of brains running an important system are replaced by a single machine learning algorithm running in isolation?

Yeah, and around 800 people a day die in auto accidents in the US. Bringing up cars bolsters my point, which is that "we rely on the brain without understanding it and that works out fine" is not a good argument. So let's be careful. I'm mystified as to why that seems to be controversial.

Are you saying that we should require 2 drivers per car? It seems like at the moment the risk/reward ratio of allowing people to drive a car after getting a license seems to work out well enough. 1 death per 100 million miles driven, and falling.

No. I'm saying

> that "we rely on the brain without understanding it and that works out fine" is not a good argument.

It wasn't my idea to bring cars into it particularly.

one would hope so!

the depressing thing about machine learning to me is the many convincing explanations of phenomena, that then turn out not to explain things (all the explanations of why dropout is effective, or why there are adversarial examples, things like that.)

this is where the comparison to alchemy is the sharpest in my opinion. The alchemists had extremely sophisticated theories that they used to provide explanations for the phenomena they observed; just ultimately none of them made any sense.

I think people end up reaching for straws because a lot of this stuff works well in practice and they’re trying to make sense of it. 20 years from now we’ll have a much clearer picture of why the techniques work! Until then, a lot of the explanations, maybe even some valid ones, may sound like baloney!

ML is a souped-up version of "draw a line through these points". We can get really efficient at drawing lines through points, but it's not like we'll suddenly realize some deeper fundamental theory about it!

But there actually is a huge amount of theory behind that problem. You can exactly derive the method that finds the best line. You can get error bounds on each of your coefficients and confidence intervals for them. You can alter the strength of your assumptions (e.g. about distribution of errors, homoskedacity, and so on) and see how it affects your model. You can add L1 or L2 regularization, both of which also have solid theoretical grounding. And so on.

All of these things help make your model more robust and give you greater confidence in it, which will be important if we want to put ML in, say, healthcare or defense. But you don’t get as much of this theory with more complex ML models, and certainly not with neural nets. Good luck trying to get a confidence interval for the optimal value of a weight in your net, much less interpreting it.

The reason you don't expect to see a deep fundamental theory of drawing a line through a few points is because you can always do it. ML doesn't always work, and sometimes it is harder to get working than other times. What's going on?

You can always draw a line through the points, but it isn't always a good approximation. If the points are inherently bunched around a line, then a line through them will approximate them well. If they're a big random cloud, then the line won't. It's the exact same way in ML, except the points are in n-dimensional space and "line" is replaced by "higher-dimensional curve or manifold". Sometimes (e.g. in image processing), the n-dimensional points are inherently bunched around a curve or manifold of the form you're using, and then ML works great. There's nothing deeper going on!