The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.
Put another way: you can treat an algorithm or routine as a black box if you know it works and understand its context in your system. You can even learn the right questions to ask if you do need to dig into that particular box at some point. Without some abstractions, we'd all be constantly paralyzed.
(edit: formatting)