(For predictive models) evaluate the output on a test set? Depending on what you're trying to do, the contents of the black box may not matter so very much if the results on an agreed test set are good (...and you're confident it's independent from any training data...)

Manual eyeballing and sanity-checking of output? (in my experience important however many layers of automated testing you're using, and undervalued by people who focus on software as an engineering process more than an art-form).

...and, circumstantially, probably a bunch of others. I don't think this is a field where making lots of rules is especially helpful (except, perhaps, the "always eyeball" one...)