"The list of things we want to optimize for is finite, there is some ordering to it, and the impact of various actions on those things is, in principle, calculable."

I think you might be surprised at the diversity of answers you get for this, particularly if you actually get people's unfiltered opinions and not what they think you want to hear. Try it, sometime - ask 20 random people (not connected to yourself) what their biggest problem is and what they most care about in their life. You'll get some commonalities (earning a living is a big one, as is social acceptance & validation), but as soon as you start digging into how do you wish to earn a living and what do you consider an ideal world, it very rapidly diverges.

"It's only when people start pushing their stupid agendas and start putting their personal interests in focus that the whole thing gets "difficult"."

That's called "being human". What would you rather they do, defer to your agenda?

>The list of things we want to optimize for is finite, there is some ordering to it, and the impact of various actions on those things is, in principle, calculable.

I don't think it's calculable, even in principle. Except in trivial issues, people can (and do) disagree on very fundamental ideas about what we want to optimize for and which direction we should optimize them.

>It's only when people start pushing their stupid agendas and start putting their personal interests in focus that the whole thing gets "difficult".

Why wouldn't the people put their "personal interests in focus"?

Whose interests they should put to focus? And says who?

"""There is some model for "justified" that you have, and the data either lets you justify welfare or does not (and tells you the degree to which it is so, if your thinking process is worth anything)."""

Your mistake is assuming the data is objective in the first place, and doesn't, a) contain the societal bias at the time it was written, b) contain the bias of the authors that made the study, and c) contain any inherent bias in the metrics used.

Of course the data may be biased. But as I said, reality has to add to a coherent whole (that's basically an axiom of reasoning without which we may as well pack our bags and go back to living in caves) - so with enough data, an AI should be able to filter out the bias.

You assume that all of the biases will eventually cancel out. This is not necessarily correct, especially in the case of societal bias -- an example of this is racism, where everyone went out of their way to find reasons why PoC were inferior to white people; we know this to be false (At least, the majority of the world does. America is still having some problems with this), but still have hundereds of years of biased data to the opposite effect, because of societal attitudes. This trap falls hand in hand with the fact that large amounts of scientific data is not being published (Example, clinical research: [1][2][3])

All the AI knows is the training set. This is given a necessary bias by the researchers in the process of creating the AI. The best an AI can do is exaggerate the biases that it learned from the training set (Which is, after all, the main point of using AIs -- to exaggerate favorable biases to produce data that the researches believe is valid).

[1]: www.badscience.net/2009/10/and-now-nerd-news/ [2]: https://www.researchgate.net/topic/Negative-Data-in-Science [3]: http://www.nature.com/news/half-of-us-clinical-trials-go-unp...

>(At least, the majority of the world does. America is still having some problems with this)

The rise of far-right politicians in Europe indicates that America is not alone here.

>But as I said, reality has to add to a coherent whole

External reality might. Reality of what people want, what we think is best, etc, doesn't.

And even the external reality part is close to impossible to access and convey to others whether it "adds to a coherent whole" in all situations or not.