Will it matter whether the AI is "really" intelligent when companies that implement the black box to spec thrive, while the companies that insist on having a human understand everything fall behind and fail?

Or think of the “uncanny valley” in 3D graphics -- most people can tell when a rendered image looks slightly off somehow, but most people can’t pinpoint the exact problems. “Something about the eyes just isn’t quite right”, etc.

But experience people can. I've worked at a 3D animation studio and we had this 'old guy' who started his career as a classically trained artist and sculptor. He could look at a character model that we all agreed felt a bit 'off' and point out that the problem is how you'd modeled the muscles connecting the shoulders to the arm. Hell sometimes he'd look at a model that we all felt was fine and suggest tweaking the bridge of the nose and that made it much better.