In banking and financial services we have regulations that require "model validation". Any automated decision needs to be tracked and compliance needs to check the decisions to see that they are sound. They were required after the financial crash in 2008 in order to get a better handling on financial risk modeling, but the laws were written broadly enough that they apply to other automated decisions in financial institutions. We could take lessons from these regulations and apply them elsewhere.