Processes initiated by data scientists during the execution of their role will tend to fail silently. What is meant here is, throwing an inappropriate model at otherwise good data produces unreliable (catastrophic in certain situations) results, but produces results nonetheless. Without the proper discernment of the reliability of the results, we have an unequivocal failure to execute the role. This is the oft-unmentioned companion to, but decidedly more insidious than, the "garbage in, garbage out" (i.e., right model, wrong data) aphorism.
It is up to the person performing this operation to deduce whether or not the conclusions are trustworthy. I don't see how someone can be confident of this without either relying on a pre-defined workflow verified by someone else qualified to assess the consequences, or to have those qualifications themselves.
What follows is a contrived example, but illustrative of the problem:
Consider e.g. user privacy: it is by now well-known that e.g. embedding vectors (or even merely the relationships between them) can leak a lot of information about the person or object it represents. It is not enough to understand how the forward pass of such a model commences, but also what is stored in those representations, which, having gone through a master's with quite a few people who now call themselves data scientists, I am not confident is commonly understood.