I love my bank but hate their fraud detection (someone suggested last time I brought this up that it might be outsourced).

I went way across town to save a couple hundred bucks on a new TV. By the time I got home they had flagged by card. Oh, yeah, TV. I should call them and tell them it was me.

No. They flagged it because I spent <$10 at a carwash on the way home.

My old bank would flag me for buying gas on a road trip I took 3 times, every time. When your bank thinks that going out of town is exceptional behavior, well, it feels a little judgemental.

Back when I bought a 250cc motorcycle, my card was constantly getting flagged (lots of $1.50 gas stops ~50 miles from home). It took months for this to let up.

The road trip one is more common to flag for fraud; if you leave your card at a gas station accidentally, someone may else may pick it up and make a few minor purchases. It happened to me when I used my credit card to fill up two cars - mine and my friends'. According to the attendant, it happens a lot.

True, but you also aren't tasked with going in and manually adjusting weights to improve the predictions. And if you are curious enough, you can certainly look at the theory and mathematics that underpin how these systems do work, even if you don't understand exactly how the training data is making its way into your model: someone clearly figured this out and that's why these systems work.

Put another way: you can treat an algorithm or routine as a black box if you know it works and understand its context in your system. You can even learn the right questions to ask if you do need to dig into that particular box at some point. Without some abstractions, we'd all be constantly paralyzed.

(edit: formatting)

The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.

The author has a followup post about that exact case and experiences trying to understand them: https://blog.nelhage.com/post/systems-that-defy-understandin...

Another point is that it's more dangerous to use deep-learning methods like this in highly regulated and sensitive environments like the healthcare industry for medical diagnosis. Medical devices involved in patient diagnosis are required to be transparent in its decision process in order to understand how it came to its decision. Despite CNNs having highly accurate results, its pretty much as useful as a black box to a doctor or medical professional if it cannot explain itself, which is one part of the hype of the deep-learning approach and is an ongoing problem in the AI research.

There are biases that can occur in CNNs which can skew some results and worst of all, debugging these CNNs or deep-learning systems around "what went wrong" will be harder given its black box nature and will be done by tweaking the hyper-parameters to get a "good enough" result which is unacceptable to professionals in mission critical industries or those dealing with human life.

It's an open (and kind of fascinating) field of research on how to make these more discoverable. Example: https://finale.seas.harvard.edu/files/finale/files/regional_...

There are numerous research results about building understandable, analyzable neural network models.

Most models don't bother, because that isn't a necessary step to get reasonable answers. But it's possible.

When the model is applied to a regulated part of the market, it's becoming apparent that the institutions using the model have to be able to defend it.

David Heinemeier Hansson (DHH) posted an interesting thread to Twitter after he was granted a credit limit 20x higher than his wife and the credit card company's initial response was it's just the algorithm.

https://twitter.com/dhh/status/1192540900393705474

Really? That's encouraging. I thought we're heading into some ironic dark enlightenment horseshoe of blackboxiness where the only artificial intelligence that works is as inscrutable as the natural intelligence it was meant to compete with.

The author has written about systems that defy detailed understanding as a follow up to this article in which he mentions Deep Reinforcement Learning Models as an example. It's a good read, especially in combination with this one.