In lots of business contexts, probably most, reducing variance is much more valuable than reducing mean expenses. Variance can halt downstream production, so the loss can be some huge amount of opportunity. And variance propagates through a supply chain, so your customers will hate variance in your output, as it may mean they have to ship the variance forward, which their customers hate. Plus if you allow your supply chain to get away with inconsistency, they can start to rob you with lower average quality and it will take you time to notice.
If a company has been bothering to do some process manually and they haven't outsourced it to the cheapest humans possible, then they care more about low variance than low cost. Pitching these businesses a solution that lowers cost at the expense of unknown high variance is very unattractive. Instead, you want to tell them "I can reduce your variance even further! Here's what that would cost".
But the argument still applies then - if only one incident occurs where a catastrophic mistake was not spotted by the model failsafe, and if later investigations show that the mistake could've been easily spotted by a second human, a human will be installed as a failsafe for the failsafe.
I think the general problem is the following: if a human makes a grand mistake, it can usually be attributed to a temporal lack of care, or just random bad luck, or happened because the person was having a bad day, or... it is also generally understood that making a grand mistake will be such a shock for the person responsible that this person will most likely never make the same mistake again. On the other hand, if a machine makes a grand mistake, the intuition of the general public, trained by centuries of experience with techology, is that this machine will make the same mistake again and again and again, when prompted with the same input. If the model is not designed to learn from mistakes in production, this will of course actually be true.
That is frequently true. Take the job of a lifeguard for instance. A single mistake can be catastrophic, and yet we know that people have trouble staying completely focused for hours on end.
AIs have no trouble staying focused and they can be trained to spot drowing swimmers pretty well even on a crowded beach.
Having a human lifeguard plus an AI that alerts the lifeguard when it spots something suspicious could lead to better outcomes than employing two lifeguards.
I disagree. I think you'll see the same thing as with drivers falling asleep while Tesla autopilot is running. The lifeguard will let the computer do all the monitoring since a low false negative rate combined with a low incidence rate means that most bodyguards will experience the computer being 100% reliable for weeks or months at a time. In fact it's not unreasonable that if the lifeguard sees someone drowning but the computer doesn't register it as such the lifeguard may question their own judgement based on experience.
Well, that is kind of the problem with AI. How do you fix a "bug" (if you can even call it that) in a model you cannot fully understand? Do you re-train it on the catastrophic mistake and somehow give it more weight? How can you be sure that this won't lead to any problems where previously there were none? How do you explain to a customer that your model now doesn't make mistake A anymore, but now mistakes B and C frequently occur? The only safe bet is to write some auxiliary code, which first uses the AI as a black box, and afterwards explicitly checks the result for this particular mistake. If this happens again, and again, and again, you need a human to maintain and extend this auxiliary code and also adjust it to changes in the underlying model, at which point I am quite certain just using a person of average intelligence instead of AI will be cheaper, more reliable and more flexible.
Surely this will depend on the cost of failure, and the cost of the human failsafe.
Spellcheck in an email client helps prevent the minor embarrassment of typos and spelling errors in emails, and few emails are so consequential that it's worth having them carefully manually vetted.
Let's also not forget that most business processes are bespoke to each corporation. Finding a single process that can be successfully targeted across multiple companies is hard. Some people like AWS and Google are trying with Textract and some of these other AI-as-a-service products, but they're not having a lot of success. They still fail all the time.
I think this is the core of the problem. 99% isn't good enough. Even 99.9% isn't good enough when we are talking acceptable accepted error margins. Even if humans make more mistakes than the AI, telling our customer it was a human error is much easier for our customers to accept than telling them it was a program error without our threshold tolerance.
We see this with self driving cars. People's reactions to the machines is that the machines have to be far better than humans before humans will be okay with the risks involved. This also holds for financial aspects. Imagine your grocery store telling you that there is a X% chance of being double charged for an item and that is within acceptable error tolerance. Even if X is lower than the rate that human grocers accidentally double charge will people be okay with that as the planned error rate or will they demand perfection?
Most of these issues aren't about how many 9's of reliability there are, but whether or not some person is accountable. AI is not itself accountable, only the person in charge of it.
Many companies already give gift cards or whatever when a human mistake happens as a form of customer service, and there's not enough compensation around AI error rates to make them palatable.
The veracity of this statement greatly depends on whose mistake you are saying it was.
List of things by how hard they are to say (easiest to hardest):
1) Someone else messed up. We'll deal with him.
2) Our code is bad. We'll fix it.
3) I messed up. I'm sorry.
When humans are wrong, the business’ “ego” can be saved by blaming the employee who made the call, sometimes firing them. But the process goes on with the same error rate.
But when software makes the wrong call, it feels like the business itself has done the wrong thing. With no way to externalize the blame for the decision, the blame gets placed on the decision to use ml in the first place.
It has nothing to do with accuracy or the repetitiveness of the mistake. It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes, because most companies are designed to absolve everyone at the top and blame everyone at the bottom.
I've worked in an industry (prop trading) where the costs of mistakes were real, and it is the most heavily automated industry there is. I've also worked in another part of the finance industry (hedge funds) where mistakes are paid for in other people's money, and it's super obvious that managers there are way more interested in passing off blame than fixing problems relative to how interested they are in doing these things in prop trading, and there is correspondingly less automation.
It's not about people in the population at large believing that machines are more prone to repeating mistakes than humans. Look at how many people are excited about the concept of self driving cars. It's about managers trying to convince themselves that this is true, so they can go on being unethical.
You can't fire a crappy AI and replace it with one with better judgement. If it does something weird or bad or inexplicable, you can't ask why it did that and tell it to how to be better in the future. Blame is a tool that organizations and people; you can't blame AI because you mostly can't teach AI.
AI also exposes the possibility of systemic error where humans would be stochastic.
A human might only identify the right number of rentable units from a spreadsheet (to pick an example from this article) 97% of the time when an AI might do it 99% of the time, but even the same human will have a different 3% error on each day. The consequences of failure are more limited and more dilute.
On the other hand, the AI may work perfectly right up until a holding company redesigns their data tables for the 100th time, whereupon it misreads every financial report with much more concentrated ill effect.
It's not like Google blocking your email or YouTube account, we're taking about your friggin bank account here.
I don't know how they're still in business and growing with such a process in place.
Undoing bad automation in finance often means reversing a very cheap edit on some database and pissing off a customer who’s so powerless that it won’t affect your business anyway.
I doubt its customers are generally aware of the AI issue, or if they are, they must assume it is not out of the ordinary.
they might have a high valuation, but thats about all the financial success they have.
Are they?
For example, Amazon Seller Central, Google, Youtube, can "outsource" their customer service to AI, because they are pretty much the only players, so customers have to suck it and deal with the frustration of a terrible experience by not getting help and not talking with a human.
With any other business if they get automated replies that don't solve a customer issue and they are unable to talk with a human, 99% of them say "fuck it" and go somewhere else.
This is one small realm of the relationship of AI and businesses. Then you have employees, suppliers, supply chains, finances, internal processes, so many multi-variable, fragile and nuanced systems that I doubt if they're not developed in-house, they'll probably do more damages than solve problems.
I can only speculate, but while it's true humans can fail when following a process, it's also true they can sometimes spot a potential failure even then there is no process to prevent it. They can come up with new processes, or ways to improve existing ones. They can also account for their actions, and managers can account for the activities of their team. All of this builds trust that the process can be improved in ways that can be well understood.
People are also scalable. You can implement controls like four-eyes on changes and critical metrics, so you're less exposed to one person's idiosyncrasies. It's hard to do that with AI.
Think of a DL powered, visually guided robot vs. a human on a production line.
One task might be to do QC inspection at the end of the line. Suppose something gets on the camera lens. In general, the DL system will keep chugging along and the accuracy will degrade. The human will notice this and clean his glasses.
If he sees something ambiguous as it passes on the line, he might give it a bit more attention or adjust the angle he's looking at it from. If he sees a series of the same anomalies, he will notice a pattern. Perhaps one of the machines up the line from him has started to introduce a new type of defect.
Suppose in assembly a worker has a sore muscle. He might adapt his motions to compensate, slow down, go to the doctor, take some pain pills, or take a day off. Unless programmed or trained to detect this, a robot will keep driving its failing motor harder until it breaks.
Like with self-driving cars?
Pharmacy - adding 1% more of a chemical compound to a pill could be lethal, but not otherwise detected for 3-12 months (pills sit on shelves for a long time). Figuring out why that happened will be very difficult.
Manufacturing - molding plastic parts requires very specific mixes of chemicals. Too much of one and it becomes brittle under certain temperatures, too little and it warps down under the warranty period. So if you manufacture the mold of a baby car seat, and you add or subtract 1% of a chemical, that car seat could break / shatter when it's involved in a collision. Terribly large lawsuits would occur because some C-level someone adopted a process to save headcount and reduce oversight.
Business processes are a lot like software - the first 80% takes 20% of the time and the remaining 20% takes 80% of the time.
"... and critically: there's no one to hold responsible for getting it wrong."
Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.[2]
1. "What's more, artificial research teams lack an awareness of the specific business processes and tasks that could be automated in the first place. Researchers would need to develop an intuition of the business processes involved. We haven't seen this happen in too many areas."
2. Including the ones who designed the "AI" system.
Because whoever does achieve the next unlock - should it happen - will receive an unimaginably large windfall. This is the classic intent of venture capital. In fact, I'd suggest that AI is actually one industry where VC is doing what it does best: taking extremely risky bets with a large potential upside.
> Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.
Presently, this seems to be one of its large detractors. If I have an employee do something stupid, I can say that an employee did something stupid. People might wonder why they were allowed to do that stupid thing, and what we're going to do to prevent it from happening again, but the explanation of the source is satisfactory. We're fallible, and we understand the fallibility of others (generally speaking).
AI is not that at all. If my automation does something stupid, I still have all the blame, and yet I have nowhere else to pass it off to. "We don't understand why our AI did this really stupid thing" is, frankly, not a satisfying response (nor should it be). Businesses employing AI certainly are not absolved of any form of accountability, and are arguably exposed to more of it (since they're not able to pass the blame on to another fallible human, and have to take direct accountability of a system they built but don't fully understand).
And even if the inputs are great, it's hard to understand what we can do with a 99% confidence prediction. The next great unlock that society is waiting for in the AI realm applies to industries where failure is not an easily acceptable outcome.
Take self-driving cars for example. Even if we can objectively prove that current AI models can drive on current roads in current conditions with a lower fatality rate than humans (this is debatable anyway, but let's assume) - what do we do when it knows it's not confident? If we assume the driver hasn't been paying attention during the ride so far, and a scenario comes up that the AI is uncertain about...the human now has likely mere seconds (at most) to capture their surroundings, analyze the risk, and take corrective action. If we assume the driver has been paying attention during the ride, then what was the point of the AI? Moreover: what if it thought it was confident, but it was still wrong. Who do we blame? How do we mitigate future instances of it? What were once societal problems we could blame on a fallible humans are now obscure technology problems we can't introspect. That's a scary place for a lot of people to be.
Basically, we've reached the level of AI where it can be used as a backup to humans and protect us from royally fucking something up. But the next big unlock will come when it can be the primary actor. It's hard to imagine how we'll get there without the AI actually understanding what it's processing in some real way.
Edit: rephrased the 3rd paragraph.
There are plenty of places where human validation shouldn’t be necessary, but is.
Wouldn't this itself be error prone? The operator is, after all, human.
This article doesn’t mention a really, really straightforward factor for why AI hasn’t invaded these domains despite billions of dollars being dumped into them.
An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. This immediately destroys the upside as now you’re 1) doing the process manually anyway and 2) fighting the automated system in order to do so.
99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong.