It’s not just when there are “special requests.” (Or rather, it could be, if every entity in the system were able and expected to contribute “special request” inputs — but usually there’s no place for most entities to do this.)

I have a great example that just happened to me yesterday.

• I signed up for a meal-kit service. They attempted to deliver a meal-kit to me. They failed. Repeatedly. Multiple weeks of “your kit went missing, and we’re sorry, and we’ve refunded you.”

• Why? The service apparently does their logistics via FedEx ground, though they didn’t mention this anywhere. So, FedEx failed to deliver to me.

• Why? Because the meal-kit service wants the package delivered on a Saturday, but FedEx thinks they’re delivering to a business address and that the business isn’t open until Monday, so they didn’t even try to deliver the package, until the food inside went bad.

* Why did FedEx think this? Well, now we get to the point where a computer followed a rule. See, FedEx is usually really bad at delivering to my apartment building. They don’t even bother to call the buzzer code I have in Address Line 2 of my mailing address, instead sticking a “no buzzer number” slip on the window and making me take a 50min ride on public transit to pick up my package from their depot. But FedEx has this thing called “FedEx Delivery Manager”, which you can use to set up redirect rules, e.g. “if something would go to my apartment, instead send it to pick-up point A [that is actually pretty inconvenient to me and has bad hours, but isn’t nearly as inconvenient to go to as the depot itself].”

I set such a redirect rule, because, for my situation, for most packages, it makes 100% sense. And, I thought, “if there’s ever a case where someone’s shipping something special to me via FedEx, I’ll be able to know that long in advance, and turn off the redirect rule.” But I didn’t know about this shipment, because the meal-kit service never mentioned they were using FedEx as a logistics provider until it was too late.

Some computer within FedEx automatically applied the redirect rule, without any human supervising. Once applied, there was no way to revert the decision—the package was now classified as a delayed ground shipment, to be delivered on Monday. (Apparently, this is because the rule gets applied at point of send, as part of calculating the shipping price of the sender; and so undoing the redirect would retroactively require the sender to pay more for shipping.)

A supervising human in the redirect-rule pipeline would easily have intuited “this is a meal-kit, requiring immediate delivery. It is being delivered on a weekend. The redirect location is closed on weekends. Despite the redirect rule, the recipient very likely wants this to go to their house rather than to some random pick-up point that we can’t deliver to.”

You get me? You can’t teach a computer to see the “gestalt” of a situation like that. If you tried to come up with a sub-rule to handle just this situation, it’d likely cause more false negatives than true negatives, and so people wouldn’t get their redirect rules applied when they wanted them to be. But a human can look at this, and know exactly what implicit goal the pipeline of sender-to-recipient was trying to accomplish by sending this package; and so immediately know what they should actually do to accomplish the goal, rules be damned.

And if they don’t—if they’re not confident—as a human, their instinct will be to phone me and ask what my intent is! A computer’s “instinct”, meanwhile, when generating a low-confidence classification output, is to just still generate that output, unless the designer of the system has specifically foreseen that cases like this could come up in this part of the pipeline, and so has specifically designed the system to have an “unknown” output to its classification enum, such that the programmer responsible for setting up the classifier has something to emit there that’ll actually get taken up.

> You can’t teach a computer to see the “gestalt” of a situation like that.

That's not necessary. You can teach a computer to recognize anomalies and route those to humans. Repeated failures is an obvious one.

> A computer’s “instinct”, meanwhile, when generating a low-confidence classification output, is to just still generate that output

That's a poorly designed system. Human failure.

All systems are poorly designed. There is no perfect system. But the default failure state of AI not predictively accounting for a case is making a bad decision; while the failure state of a “cybernetic expert system” not predictively accounting for a case is stalling in confusion and asking for more input. Usually, stalling in confusion and asking for more input is exactly what we want. You don’t want an undertrained system to have false confidence.

If we could get pure-AI systems to be “confused by default” like humans are, such that they insist on emitting “unknown” classification-states whether you ask for them or not, they’d be a lot more like humans, and maybe I wouldn’t see humans as having such an advantage here.

I don't know what you mean by "pure-AI systems." I work in this field and have many times implemented a review in the loop, or a route for review. It's an old technique, predating computers.

https://en.m.wikipedia.org/wiki/Dead_letter_mail

A "pure-AI system" is a fully-autonomous ML expert system. For example, a spam classifier. In these systems, humans are never brought into the loop at decision-making time — instead, the model makes a decision, acts, and then humans have to deal with the consequences of "dumb" actions (e.g. by looking through their spam folders for false positives) — acting later to reverse the model's action, rather than the model pausing to allow the human to subsume it. This later reversal ("mark as not spam") may train the model; but the model still did a dumb thing at the time, that may have had lasting consequences ("sorry, I didn't get your message, it went to spam") that could have been avoided if the model itself could "choose to not act", emitting a "NULL" result that would crash any workflow-engine it's embedded within unless it gets subsumed by a non-NULL decision further up the chain.

Yes, I'm certain that training ML models to separately classify low-confidence outputs, and getting a human in the loop to handle these cases, is a well-known technique in ML-participant business workflow engine design. But I'm not talking about ML-participant business workflow engine design; I'm talking about the lower-level of raw ML-model architecture. I'm talking about adversarial systems component design here: trying to create ML model architectures which assume the business-workflow-engine designer is an idiot or malfeasant, and which force the designer to do the right thing whether they like it or not. (Because, well, look at most existing workflow systems. Is this design technique really as "well-known" as you say? It's certainly not universal; let alone considered part of the Engineering "duty and responsibility" of Systems Engineers—the things they, as Engineers, have to check for in order to sign off on the system; the things they'd be considered malfeasant Engineers if they forget about.)

What I'm saying is that it would be sensible to have models for which it is impossible to ask them to make a purely enumerative classification with no option for "I don't know" or "this seems like an exceptional category that I recognize, but where I haven't been trained well-enough to know what answer I should give about it." Models that automatically train "I don't know" states into themselves — or rather, where every high-confidence output state of the system "evolves out of" a base "I don't know" state, such that not just weird input, but also weird combinations of normal input that were unseen in the training data, result in "I don't know." (This is unlike current ML linear approximators, where you'll never see a model that is high-confidence about all the individual elements of something, but low-confidence about the combination of those elements. Your spam filtering engine should be confused the first time it sees GTUBE and the hacked-in algorithmic part of it says "1.0 confidence, that's spam." It should be confused by its own confidence in the face of no individual elements firing. You should have to train it that that's an allowed thing to happen—because in almost all other situations where that would happen, it'd be a bug!)

Ideally, while I'm dreaming, the model itself would also have a sort of online pseudo-training where it is fed back the business-workflow process result of its outputs — not to learn from them, but rather to act as a self-check on the higher-level workflow process (like line-of-duty humans do!) where the model would "get upset" and refuse to operate further, if the higher-level process is treating the model's "I don't know" signals no differently than its high-confidence signals (i.e. if it's bucketing "I don't know" as if it meant the same as some specific category, 100% of the time.) Essentially, where the component-as-employee would "file a grievance" with the system. The idea would be that a systems designer literally could not create a workflow with such models as components, but avoid having an "exceptional situation handling" decision-maker component (whether that be a human, or another AI with different knowledge); just like the systems designer of a factory that employs real humans wouldn't be able to tell the humans to "shut up and do their jobs" with no ability to report exceptional cases to a supervisor, without that becoming a grievance.

When designing a system with humans as components, you're forced to take into account that the humans won't do their jobs unless they can bubble up issues. Ideally, IMHO, ML models for use in business-process workflow automation would have the same property. You shouldn't be able to tell the model to "shut up and decide."

(And while a systems designer could be bullheaded and just switch to a simpler ML architecture that never "refuses to decide", if we had these hypothetical "moody" ML models, we could always then do what we do for civil engineering: building codes, government inspectors, etc. It's hard/impractical to check a whole business rules engine for exhaustive human-in-the-loop conditions; but it's easy/practical enough to just check that all the ML models in the system have architectures that force human-in-the-loop conditions.)

> humans have to deal with the consequences of "dumb" actions (e.g. by looking through their spam folders for false positives)

Email programs generally have a mechanism for reviewing email and changing the classification. I think your "pure-AI" phrase describes a system that doesn't have any mechanism for reviewing and adjusting the machine's classification. The fact that a spam message winds up in your inbox sometimes is probably that low-confidence human-in-the-loop process we've been talking about. I'm sure that the system errs on the side of classifying spam as ham, because the reverse is much worse. Why have two different interfaces for reading emails, one for reading known-ham and one for reviewing suspected-spam, when you can combine the two seamlessly?

Perhaps you've confused bad user interface decisions for bad machine learning system decisions. I'd like to see some kind of likelihood-spam indicator (which the ML system undoubtedly reports) rather than a binary spam-or-not, but the interface designer chose to arbitrarily threshold. I think in this case you should blame the user interface designer for thinking that people are stupid and can't handle non-binary classifications. We're all hip to "they" these days.