The first thing I can think of that was like the 'AI' we have now is seeing the cubic spline method. It was shockingly good at particular points. But between those points it could be 'meh close enough' or wildly bad. AI we have now seems more along the lines of 'meh close enough'. Which for many things is actually good enough. For me I have always looked at the nets as a cubic spline smeared across dozens of nodes with a newtons method for searching out 'optimal'. Bad analogy I know. But that is the closest I can get to explaining 'how they work'.

The other factor here is that we appear to be trying really hard to keep it "magic" wrt ourselves. At the very least, the notion that it might be "just computation" in the brain seems to be very discomforting to some people.

Which might affect the way we look at technologies that try to model it, or at least use similar approaches - i.e. we're primed to treat them as more "magic" than they really are, or else to invent some reason why we think ourselves to be different.