The problem with black boxes is that they beg parallel construction theories.

I.e. "We don't approve loans to {ethnic majority} on purpose. We only use our models, which only rely on historical data." -> No proof, no case, no problem

The point of any AI law should be to put the onus on the model executor to explain how their model doesn't violate existing law.

You can use any model you want, but you can't point to its unexplainability as a free pass for ignoring protected classes.

And if you can't explain its results? Well, then you're opening yourself up to lawsuits.

Companies can already do that without AI. Why would you legislate just AI and let some other company discriminate?

Why do you think they would let?

So ML systems are guilty until they can prove themselves innocent?

Requiring the claimant have some evidence might not be a perfect legal regime, but the converse is a de-facto ban on innovation.

Can a human _prove_ that their decisions are made without bias? No. Are we going to hold them to the same standard proposed here for an AI? Also no. In practice, rules like the one proposed here just enshrine legacy decision systems which are already understood to biased and ineffective.

The most mysterious black box we have is the human brain.

A fair point.

What would you say of requiring training data and hyperparameter archiving for subsequent legal review?

The original regulation doesn’t actually sound that vague. From TFA

> The regulation, proposed by the Commission in 2021, was designed to ban some AI applications like social scoring, manipulation and some instances of facial recognition. It would also designate some specific uses of AI as “high-risk,” binding developers to stricter requirements of transparency, safety and human oversight.

The issue is that as written it would still not cover general AI and that would still be legally grey.

Some of those uses in the US have already been problematic, like Madison Square Garden using computer vision to ban any lawyers from firms suing them from their vast properties, which is actually against various laws like their liquor license.

That's still focusing on the wrong problem though: if Madison Square garden had simply told security to eject the lawyer, it would be the same issue. The problem isn't facial recognition, it's that they thought they could or should do this at all.

Not entirely. ML algorithms make these processes ridiculously effective. You no longer need to expend the time of a human to make that decision.