The prosecutors are the "they". The further removed you get from people who know the 'AI' the more the validity of the evidence provided by the AI relies merely on what others have said.

The people working on the system know that it can be faulty. Other software developers have an idea that it could be faulty. Prosecutors are told that it's alright. They turn around and argue it in a court of law. And the further you get the less people have any idea that the system is faulty. Down the line people will go "the machine says so, therefore it must be so".