I'd be a lot more comfortable if metaethics and related philosophy advanced to the point where we didn't have to rely on indirect normativity. But I don't think this is something you can just throw money and people at, because this is a field which is sufficiently difficult and hard to judge, that work which isn't first-rate tends to detract from progress by adding noise rather than help.
Oh god no. A thousand times no.
Humans are masters at hiding sociopathy behind cultural and political obfuscation.
A machine that looks at human leaders and generalises human values from them is absolutely guaranteed to be a monster.
>I'd be a lot more comfortable if metaethics and related philosophy advanced to the point where we didn't have to rely on indirect normativity.
We already have any number of perfectly sustainable ethical systems. What we don't have is the ability to follow them in practice - partly because all political systems tend to reward power and influence with more power and influence, and partly because even when there are token checks and balances (e.g. in the form of representative democracy) it's so easy to game them with appeals to irrationality and emotion.
This rarely works out well for most of the population.
But I don't much like the content of the proposal. I think you can have extremely useful intelligence amplification without any need for agency or personality.
Much better modelling along the lines of 'If you do this, this will be the outcome' would be a game changer on its own.
Humanity has already solved ethics, so there's no reason for us to give up and ask AI to define ethics for us.
That's the point...
>Humanity has already solved ethics
Where on earth do you get this idea?
Here's a relevant essay http://medium.com/@yanazendo/how-to-be-lazy-and-successful-a...
The simplest monoid combines ethical systems by only including codes that appear in both Codes.
So, the ACM+IEEE Code of Ethics can simply be defined as the intersection of the ACM+IEEE Code of Ethics.
Then ACM knows what to expect from IEEE, and vice versa.
If we want AI to be ethical, we must be able to express ethics computationally, and we must be able to formally verify the correctness ethical systems.
So, we should use monoids.
If the research is aimed at ensuring that AI is aligned with human interests, another priority should be researching what we deem those interests to be.
The Law and Ethics research section of the research priorities document is centered on mechanisms, with a couple of nods to policy, but with the seeming assumption that that question of what is in the human interest has already been answered.
But the answer certainly isn't obvious. Different sets of humans have wildly diverging and fundamental disagreements over what is right or wrong for humans and our future.
I doubt we're going to find a consensus in the time we have. It will probably make more sense to just have smart people define a set of good principles. This is hard enough that it should be considered a research task, and I would think it would be at the top of the priority list.
Otherwise, how do we know that the research we are doing on other problems even makes sense? If you just stub out the "human wishes" component of all this as one simple "human approves" button, then you haven't solved anything. We need a robust definition of what "aligned with human interests" means.