Not quite. What I'm saying is that the explanation is valuable because it comes from the accountable party. That you have a good explanation does not absolve you of responsibility. The short and clear way to put it is this: You don't go to jail because YOU think you did something wrong, you go to jail because WE think you did something wrong.

If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process.

>If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process.

Well, a court/law just has to declare "AI" as allowed to be used in such decisions, and the whole recourse you describe vanishes though...

> my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that.

> How do they justify that AI system getting to make decisions that affect me and my life.

so you, a priori, make the assumption that your loan _should've_ been accepted?

If the decision wasn't an AI, but some actuarial that calculates and computes based on a set of criteria, and the result is a denial, you could still make the same argument of "why is _this_ the process, instead of something else (that makes my loan acceptable)?".

> I care what about how the accountable bankers justify putting that AI system into the process in the first place.

Computers are already deciding. As to why: it's because it's their money they're lending.

I work in a bank and you're laughably wrong. Computers are at best assisting in the lending process but more realistically they're completely ignored. There is NO AI (at least recognizably, you could argue that there's some sort of very simple expert model embedded in the 10 credit limits) involved in the process at all.

Out lending process is based on the judgment of a few specific individuals, with more involved clients requiring approval from more senior people. All steps of that process can be overturned by the overseeing person, and that person is accountable for their decision.

I didn't say there was AI.

Article 22 of the GDPR states “The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.”

Hidden in the definition of sufficient is "meeting the standards for absolution that an investigator would use". I honestly see no reason a system could not go so far as providing case law references or FICO + ECOA guidance as evidence for a compliant rejection, even if the underlying reason is some statistics based on sex.

And why would a perfectly reasonable bank tell its customers it's using AI? AI would provide the breadcrumbs and the loan officer would conduct a reasonable story using that - it's just parallel reconstruction at its finest. I imagine this is how credit scores work. A number comes out of the system and the officer has the messy job of explaining it.

I used to work in munitions export compliance and there intent really matters. It's the difference between a warning and going to federal prison. And intent is just a plausible story with evidence to back your decision, once you strip the emotion away.