But they can explain their 'reasoning' in a way that makes sense to humans a lot of the time. Serious question: how do you know if something does or doesn't reason?
The LLM only correlates, so it's "reasoning" is something like "most often people answered 4 to 2+2 then that I should write". That's why it gives out confidently complete gibberish as it works with correlation and not causality. I think much closer to that goal of real reasoning are world models - check out something like DreamerV3 or what Yann Le Cunn is talking about.
While I agree LLMs are unlikely to be the last word on AI, the fact we understand alignment so poorly that they spew random things, let alone any arguments about which words are acceptable[0], is a sign we have much foundational work to do.
Indeed, as I recall, one of the main researchers in this topic describes it as "pre paradigmatic" because we don't have a way to even compare the relative alignment of any two AI.
[0] personally, I suspect but cannot prove that tabooing certain words is a Potemkin village solution to the underlying social problems
It doesn't matter bout "thinking" or whatever. Any black box system will be uncontrollable in essence. You can not make inviolable rules for a system you don't understand.
And saying LLMs hallucinate because they don't understand anything is stupid. And just shows ignorance on your part. Models hallucinate because they're rewarded for plausibly guessing during training when knowledge fails. Plausibly guessing is a much better strategy to reducing loss.
And the conclusion is obvious enough. Bugger smarter models hallucinate less because they guess less. That holds true.
https://crfm.stanford.edu/helm/latest/?group=core_scenarios
All the instruct tuned models on this list follow that trend.
From Ada to Babbage to Curie to Claude to Davinci-002/003. Greater size equals Greater truthfulness (evaluated on TruthfulQA)
This is a very bold assumption that the current LLMs function and "think" in the same way some future AGI would. They do not even reason, just make up words that fit some context - thus they "hallucinate".
There is no reason the approach taken here by injecting some bias or word filtering would apply to the real thing. And AI safety and aligment is not (at least it was not until getting hijacked) and was not about some model saying mean words but something really threatening like the paperclip maker problem - an agent choosing a path to a goal which is not aligned with what humans find acceptable (e.g. solving world hunger by killing everyone)