> In other words, I don't think it's even possible to codify what the data the ML algorithms should be trained on -- they're culturally specific down to the level of an individual town in some sense; and require looking at huge populations at others.

This is inciteful. ML is by definition generalizing, and should only be used where it's ok to generalize. There is an implicit assumption in use cases like medical diagnosis that there is a latent representation of the condition that had a much lower dimensionality than the data and that the model is trained in such a way that there are no shortcuts to generalization that miss out on any information that may be important. The second condition is the hardest to meet I believe, because even if a model could take in lots of outside factors, it probably doesn't need to to do really well in training and validation, so it doesn't. The result is models that generalize, as you say, to the population instead of the individual, and end up throwing away vital context to the personal case.

I also believe this is an important consideration for many other ML applications. For example those models that predict recidivism rates. I'm sure its possible to build an accurate one, but almost certainly these models stereotypes in the way I mention above, and do not actually take the individual case into account, making them unfair to use on actual people.