It sounds like a useful test if you get a positive result for discrimination. If the test comes back negative, things are more complicated.

This isn't as simple as it seems because a supposed job requirement, while superficially neutral, may be unnecessary. For example, police departments used to have height requirements. But height requirements discriminate against women. Do you need to be tall to be a police officer? Maybe it's really about projecting authority? Okay, how do you measure that?

Making job requirements neutral is a rather fuzzy area. What is the job really about? Who decides?

Any variable that's not independent of gender can be used to (imperfectly) detect gender. It may not be easy to figure out if machine learning is using it for something real or just as a substitute gender-detector. Randomizing gender while leaving everything else the same isn't going to detect this.

Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge issues a court order to "make it stop doing that."

How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers' inability to control the behavior of a black-box AI system.

If the manufacturer explains to the judge that they don't really know what's going on inside the AI, the judge won't be inclined to cut them a break.

On the other hand, can you actually make a human not repeat given action? Or explain it?

I don't think that that is really the Problem. We also know how to teach people something, without knowing how our Human Brain works. The Problem is Companies can select for certain unlawful behaviour, but nobody can proof that later.

AI models simply exhibit the behaviour of the data they are trained on. If you make a model that tries to copy an individual persons behaviours, is it the AI model that is at fault or the person it's modelling? Where should the liability lay?

the problem with that also is that humans have emergent higher level reasoning, as a trivial, super simplified example, imagine if you trained a fictional AI on policy towards slaves before the worldwide ban on slavery, what sort of things would it suggest, its highly unlikely "banning slavery" would be one of them.