It is much harder to ask an algorithm about it's reasoning than a person's.

"Joe, why did you fire Carla?"

"She was late to work every other day and spent all day on her phone"

"HR-bot 5000, why did you fire Carla?"

"Weights indices 59738, 837, and 28836 added to over 0.67, yielding a score of only .85, which is below the firing threshold"

Certainly you could look at the input data and try to convince a jury that her performance wasn't acceptable but you can't prove that is why the AI fired her.

We cannot do this today, but this capability, of using language, is one we are actively working towards. Eventually, you can imagine an AI that can engage in a dialog like this one, in words. For example, see the DARPA Explainable AI program: https://www.darpa.mil/program/explainable-artificial-intelli....

Great link, thanks. These are the kind of systems that will really allow AI to take off.

At my previous shop, it was normal to silently tolerate developers breaking the rules (not locking their stations when they stepped away, browsing unapproved sites, coming in late, leaving early)... until it was decided that they needed to be fired because they were rude to someone important, or etc... at that point they lost their privilege to act like a professional and their badge-access records were pulled and compared to their time-entry.

The reason developers got fired is because of clear and well-defined rules... that were never enforced except when the decision to fire them for an arbitrary reason had already been made.

I think HR-bot 5000 would treat employees more equitably by forcing employers to confront the inequity of some HR regulations. The same way self-driving cars "drive weird" because they stop at stop signs and drive the speed limit.

You can also have paper trail documenting that you never did anything illicit during the development of the model.

There will be protocols for sensitive areas.

Can't do the same with individual decision making.