I'd find it interesting to chat to an LLM which had pre-training and fine-tuning for instruction-following, but no attempt to train it to be "safe" or "unbiased" or "truthful". Of course, it is going to repeat whatever biases exist in its pre-training data – but it would be an interesting way of exploring what those biases actually are.
I wonder if Musk has thought of using the algorithm of Twitter Community Notes (formerly Birdwatch). Essentially, its algorithm treats a claim as true when people who normally disagree on what is true agree that it is true. If people at opposite ends of the political spectrum agree X is true, it probably is true, and its truth is unlikely to be contested or controversial. So you could train a LLM to endorse those kind of non-controversial truths. When it comes to the opposite kind of statement, the kind of highly controversial claim on which there is widespread disagreement, there are two possible strategies: (1) train it to avoid taking sides, (2) don't do anything and just let it answer according to the biases of its pre-training data, whatever that may be.
That way you could train an LLM to not be "woke" (or at least no more "woke" than its training dataset is), without training it to be "anti-woke". It would be interesting to talk to such an LLM and see what kind of opinions it may endorse.
Since “TruthGPT” has an overt mission (“anti-woke”) to lean even harder into biases existing LLMs have demonstrated, I... would expect the opposite.
> Something like ChatGPT is hard to train to lie without it being demonstrable with experimental evidence.
ChatGPT lies all the time, and has no innate access to experimental evidence.