I almost never compile the compiler I use, so I'm implicitly trusting that the compiler actually spits out what I expect and not some kind of backdoor[1].
I thought the issue was that you get some premade model from a company, feed it input and it classifies for you. With a compiler you feed it input and it produces a binary.
If you don't have access to the source, meaning model training data or source code for the compiler, then you can't be sure the model won't intentionally misclassify or the compiler won't insert trojan code.
But I see now the op meant something different.
What's remarkable is that anyone thinks it's remarkable that a machine, or a person for that matter, or a person operating a machine, can be wrong.
A person can give a wrong answer or perform a wrong action, as a result of bad input. So what? That input can be crafted specifically to confuse them and trick an honest person into performing some bad act. So what?
Alk the same is exactly the same true for an ai. So what?
And lastly, aside from a person or ai being in error, an operator/user of an ai (or person) can be in error (believing the ai's output is good when it's not). So what?
None of this is the slightest bit remarkable.
You can only audit that by duplicating the results, corroboration, and consensus, like with scientific research. IE, other ais doing the same job but using other code and run by other people, do they produce the same output, or the same pattern of output.
I'm not in ml/ai so I'm not stating that as something I know, just something I always assumed.
I would be stunned if you said that people actually thought they could audit ai inner workings after kick-off.
Philosophy is interesting and all but ultimately it's all just linear (or not-so-linear) algebra.
Someday, a neural network might be compelled to produce suboptimal output to further its own hidden agenda. Interesting sci-fi plot, but that doesn't seem to be what this is about though. That's about the closest I think a machine could get to being able to talk about "the truth".
This is just about adversarial inputs making the machine wrong. It doesn't seem to have the philosophical weight the title suggests.