>This is basically saying that human values are complex enough that we should instruct an AI to study humans to figure out what they are, rather than try to program values in directly and risk getting them wrong.

Oh god no. A thousand times no.

Humans are masters at hiding sociopathy behind cultural and political obfuscation.

A machine that looks at human leaders and generalises human values from them is absolutely guaranteed to be a monster.

>I'd be a lot more comfortable if metaethics and related philosophy advanced to the point where we didn't have to rely on indirect normativity.

We already have any number of perfectly sustainable ethical systems. What we don't have is the ability to follow them in practice - partly because all political systems tend to reward power and influence with more power and influence, and partly because even when there are token checks and balances (e.g. in the form of representative democracy) it's so easy to game them with appeals to irrationality and emotion.

This rarely works out well for most of the population.

But I don't much like the content of the proposal. I think you can have extremely useful intelligence amplification without any need for agency or personality.

Much better modelling along the lines of 'If you do this, this will be the outcome' would be a game changer on its own.