This is also why a lot of business logic software is a lot more crappy than it seems like it ought to be. For sure some of it is written by people who don't know how to design a good software product. But there's also a lot of it that can't be made well because the people asking for it to be made don't understand (on like a logical or specification level) what their job actually is.
If machine learning is a half-arsing industry, then it's only there because every industry that machine learning is attempting to augment is also an industry that is half-arsing.
EDIT: And I think it's worth mentioning that I very well may owe my life to one of these "half-arsers" that I'm talking about. He found and removed a precancerous mole. The tests came back showing pre-basal cells (iirc). I asked a lot of questions about how he was able to identify the mole as problematic (and he saw it right off the bat even though my general practitioner and myself ignored it) and he really wasn't able to give a satisfying answer that didn't also identify other moles he said were (and have been) safe. He just knew it was bad and he was right.
You have no insight into the internal model, you cannot check or improve it. A single training mistake renders the whole model afoul.
Most of AI as I learned it in school needs internal verfification steps, you print out the inferred rules, so you can learn from them. The black-art model really is just black art, but how should e.g. a self-driving car be allowed to drive on the streets, when you cannot explain the most basic rules, in case this or that.
Instead of understanding the system you are trying to control, you train the machine with some data and hope for the best.