> Now we might have a model that fulfills (2) but we cannot be sure because we used a black-box approach.

It's not like the alternative is any different. Being sure you have the right answer is not viable, the other methods of getting there are just more transparent heuristics.