> The problem here is that it's missing the "glue" to more real world applications. This is where more humdrum software engineering comes in.

This is a bold statement. The world does not function based on "well-defined rewards". The concept of "common sense", which some consider table stakes for a human operating competently in our world, is mostly made up of things which are neither well-defined, nor allow a tremendous amount of training data. Current ML approaches require both.

Correct, but there are likely applications of AI in strategic reasoning that are used in the real world (outside the obvious finance bots) by engineering a "translation layer" from real world constraints to AI-compatible value functions and back.

But in general, yes, this is why since 2013 we don't see AI making anywhere as massive strides everywhere as they do in boxed-in applications like games.

> The world does not function based on "well-defined rewards".

Humans can be neatly reduced to a "procreate, and make sure your progeny procreate" value function. A lot of apparent counterexamples to that turn out to make sense when 2nd order effects on progeny are considered.

Humans are adaptation-executors, not fitness-maximizers. Evolution has a procreation value function; humans have procreation-favoring traits.