It would be really interesting to know what Feynman would make of the current state of machine learning. It's awesome what tasks can be performed with it, but I imagine he would be disappointed with our level of understanding of how these systems work.

Why would you say we don't understand how these systems work? Stochastic gradient descent, for example, is not particularly enigmatic. Pattern Recognition and Machine Learning by Christopher Bishop is a good place to start if you want to gain an understanding of how and why machine learning algorithms work.

He's talking about understanding their process, not their mechanics, machine learning systems are usually black boxes with no guarantees, we run into issues with this fact regularly because all we can do is train them and then study the results, it can't tell us anything for certain.

Because of emergent phenomena. That’s like saying you understand how organisms work because you understand how molecules work. Look at Wolfram’a New Kind of Science and even finite automata can have amazing patterns.