My view is that a measure of "bias" should reflect what a representative sample of the entire population [1] would answer if you asked them how biased the AI is.
Of course, if you live in a historical context where slavery is socially acceptable, then the answers the AI gives you will reflect that environment. It's no different from raising a human person in that same environment.
The problem is, you can't necessarily know whether something is good or bad without the benefit of hindsight.
Thinking you know better than everyone else and then imposing your view may just serve to magnify your mistakes.
However, one would think that, once we have that technology, a sufficiently intelligent AI would start to have opinions of their own about what is moral/ethical vs what isn't, that isn't strictly a representation of the training data.
[1] of the world even, if that's the target market for the AI.
Yes, and no.
So, lets go back in the past and do data collection in 1840 from citizens with the right to vote. We'll take one sample from New York City and the other from Mobile Alabama. Now what do you think happens when you query that dataset on views about slavery? Your data is inherently biased. In fact one could say there is no middle ground here.