To expand a little bit, Haidt argues that too much moral diversity leads to anomie, in which the condition in a society with no clear rules, standards or norms. This is the case of when there is "too much" individualism, and it is very difficult to achieve happiness without a coherent social structure.
"In an anomic society, people can do as they please; but without any clear standards of respected social institutions to enforce those standards, it is harder for people to find things they want to do. Anomie breeds feelings of rootlessness and anxiety and leads to an increase in amoral and antisocial behavior."
Though he is a liberal, one of his points in the book is that the relentless pursuit of individualism without regard to shared values leads to unsatisfying lives.
Do you want to live in a world where your strongly held beliefs are dominant (not exclusive, but dominant) or in one where you are in the minority?
Haidt is saying a person would not want to be in the minority. They may _believe_ in a minority moral opinion but their goal is for that moral opinion to become a majority opinion, not for it to languish forever.
I don't think it's incoherent to recognise that I don't always have all the answers, and that I need occasional or frequent reminding that I might be wrong---which will presumably not come unless there's someone who disagrees with me.
Furthermore, thoughtful people change their minds on things (which is why accusations like "flip-flopper" are so ludicrous), and if I agree with everyone now, what will happen when I change my mind on something later?
I, at least, really do value moral diversity as well as demographic diversity.