it doesn't have to be though, theres a lot of progress in explainable AI, causal inference etc. but a lot of software products like let's say Instagram make you have to trust their AI because they don't show you the metrics, training data or really anything at all. the company is a black box. even if they select posts to show you by Minecraft chicken randomisers you'd not know