Everything from LLMs are hallucinations. They don’t store facts. They store language patterns.
Their output semantically matching reality is not something that can ever be counted on. LLMs don’t deal with semantics at all. All semantics are provided by the user.
People use the term "hallucination" to refer to output from LLMs that is factually incorrect. So if the LLM says "Water is two parts hydrogen and one part oxygen" that is not a hallucination.
LLMs are broken in the same way. They are just predictive text generators, with no real knowledge of concepts or reasoning. As it happens most of the text it has been trained on is factual, so when it regurgitates that text it is only by happenstance, not function, that it produces facts. When it hallucinates a completely new sentence by mashing its learned texts together, it's pure chance whether the resulting sentence is truthful or not. Every generation is a hallucination. Some hallucinations happen to be sentences that reflect the truth. The LLM has no ability to tell the difference.
An LLM is doing the exact same thing when it generates output that you consider to be a "hallucination" that it's doing when it generates output that you consider "correct".
Another way of framing it would be along the lines of 'catastrophic backtracking' but attenuated: a transformer attention head veering off the beaten path due to query/parameter mismatches.
These are by no means exhaustive or complete, but I would suggest knowledge exhaustion, stochastic backtracking, or simply perplexion.
Verbiage along the lines of misconstrue, fabricate and confabulate have anecdotally been used to describe this state of perplexity.
Like, it's a bit sarcastic, sure, but until factuality is explicitly built into the model, I don't think we should use any terminology that implies that the outputs are trustworthy in any way. Until then, every output is like a lucky guess.
Similar to a student flipping a coin to answer a multiple choice test. Though they get the correct answer sometimes, it says nothing at all about what they know, or how much we can trust them when we ask a question that we don't already know the answer to. Every LLM user should keep that in mind.
Yes, it means proper application of the term means you have to know what went into its training data (or current context), but you'd have to make those assumptions anyway to be able to put any credence at all to any of its outputs.
Most people call that "thinking".
But that's exactly what philosophy has been arguing about regarding humans since at least Descartes's Evil Demon (the 17th century version of the brain in a vat). Humans don't know anything about "reality", they only know what their senses are telling them. Which is at best a very skewed and limited view of reality, and at worst completely wrong or an illusion.
We perceive the world through more facets than an LLM, but fundamentally we share many of their limitations. So if someone says "LLMs don’t store facts", I find "neither do humans" a very reasonable answer, even if its only purpose is to show that "can it store facts" is a bad metric.
Of course the more productive part to argue about is the "are facts and language patterns really mutually exclusive", which leads right into "if you had to design an efficient token predictor, would it do 'dumb' math like a markov chain, or would your design include some kind of store of world knowledge? Can you encode a knowledge store in a neural network? And if you can, how can you tell that LLMs don't do that internally?"
couldn't it literally be as simple as hard checking that the prompt is contained in a response before being sent out, if so just swap it with a "safe one"
Not every step that checks LLMs needs to be more advanced, some of them can be simple. LLMs are pattern finders but we also know how to check statically known things already.
Assuming the pre-prompts are in normal english, hard checking against tokens would match against everything.
that's to say, 1+1=2 is a well known fact, so it'd get that right, but ask it to md5sum a string that is not in any existing rainbow table, and it'd get it wrong.
I've not used GPT 4 so it might have gotten better.
"V unq'ir cynlq zhpug jvgu vg ercerfrag ohg V jnf haqre gur vacebprffvba gung PungTGC jnf abg tengure ng zngpuvfgbef."
According to the internet that decodes to
"I had've playd mucht with it represent but I was under the inprocession that ChatGTP was not grather at matchistors."
base64 of the original according to GPT3.5:
"SSBoYXZlbid0IHBsYXllZCBtdWNoIHdpdGggaXQgcmVjZW50bHkgYnV0IEkgd2FzIHVuZGVlciB0aGF0IENoYXRHUFQgd2FzIG5vdCBncmVhdCBhdCBtYXRjaGVtYXRpY2FsIGNvbXB1dGlvbnM="
Decoded with online tool:
"I haven't played much with it recently but I was undeer that ChatGPT was not great at matchematical computions"
Both get worse as the sentence goes on, but they are pretty viable for information extraction. I remember GPT4 being even better.
Same with decoding ROT13, etc.
I once asked GPT-4 to generate an SVG suitable to be used on a web page, and got back <img src="data:..."> that was a valid base64-encoded SVG file that contained what I asked.
Of course OpenAI might have a completely different prompt and do some post-filtering of the ML output to replace any mention of it with a more innocent one. But that filter would have to be pretty advanced, since many prompt extraction techniques ask for the prompt a couple tokens at a time.