Plus, these models are entirely black boxes. Even given weights, we don't know how to look at them and meaningfully tell what's happening - and not only that, but training these models is likely not cheap at all.

Stable output is how we can verify that attacks like the one described in Reflections on Trusting Trust[0] don't happen.

[0] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...