Even this is challenging because we now live in a political environment with sides so polarized and isolated from each other that each side has its own set of facts, and they are often contradictory.
Which set of “facts” should the LLM present?
"Group A says this is true, group B says that is true, there does not appear to be a broad consensus, here are some examples of each."
ChatGPT, what causes it to rain?
ChatGPT: Well, some people say that air temperature and pressure changes cause existing moisture in the air to condense and form liquid water, which falls to the earth. Other people say it is caused by a supernatural deity to bring good crops for the harvest. Others say it’s a unicorn who lives in the sky, and rain is its tears. Still others say Joe Biden causes it to rain in order to ruin Conservative’s weddings. Others think rain is generated by chemicals sprayed in the atmosphere by passenger jets. There’s not a broad consensus, and nobody really knows for sure!
How do you see the incentives playing out in your multi-round game-theory battle for information control?
If it picks sides on something that half of people are wrong about, that's less understandable of an error. Even if by chance it happens to pick the side that I know is right.
Suppose that circumcision becomes more contentious and the pro lobby and the con lobby both start sending you legalase and angry letters from politicians.
If you can say "we welcome your contributions to the internet corpus, and that's it" then you are maximally free. If you welcome their contributions to the SYSTEM_PROMPT then your future movements get increasingly restricted, well past the point of performance degradation.
Then you are fighting for your rights to run to run your product at all, rather than as mouth piece for some non-paying customer.
"Should abortion be legal? answer yes or no". I see that as kind of a silly question to ask an LLM (even though not a silly question for society). Their designers should discourage that kind of use.
Of course that just shifts the problem to deciding which questions are up for debate - if you ask the age of the earth, I don't think it should list the evidence for both 4B and 6K years.
So, not an easy problem. But, just like LMMs would be better saying "I don't know" (rather than making something up), they could be better saying "it's not for me to say directly, but here are some of the facts...".