I grant you, my print Encyclopedia Britannica is not 100% accurate. But the difference between it and a LLM is not just a matter of degree: there's a "chain of custody" to information that just isn't there with a LLM.
Philosophers have a working definition of knowledge as being (at least†) "justified true belief."
Even if a LLM is right most of the time and yields "true belief", it's not justified belief and therefore cannot yield knowledge at all.
Knowledge is Google's raison d'etre and they have no business using it unless they can solve or work around this problem.
† Yes, I know about the Gettier problem, but is not relevant to the point I'm making here.
We talk about models “hallucinating” but that’s us bringing an external value judgement after the fact.
The actual process of token generation works precisely the same. It’d be more accurate to say that models always hallucinate.
I’d be wary of generalising it like that, it is like saying that all programs run on the same set of CPU instructions. NNs are function approximators, where the code is expressed in model weights rather than text, but that doesn’t make all functions the same.
Turing machine is the embodiment of all computer programs. And then you come across the halting problem. LLMs can probably generate all books in existence, but it can't applies judgement to it. Just like you need programmers to actually write the program and verify that it correctly solves the problem.
Natural languages are more flexible. There are no functions libraries, or paradigms to ease writing. And the problem space can't be specified and usually relies on shared context. Even if we could have snippets of prompts to guide text generations, the result is not that valuable.
All the stress people are feeling with GenAI comes from the over anthropomorphisation of ... stats. Impressive syntatic ability is not equivalent to semantic capability.
Or, in other words, if a chatbot really were hallucinating, it would probably start giving unprompted responses.
> Humans can hallucinate, its a deviation from what is observable reality.
What is "observable reality" then, for an LLM? Its training set?
I don't see how that's different than the LLM case, a lot of algorithms change as a function of the data they're processing.
Inconsistent results could be resolved by introducing additional out of data constraints (e.g. incremental hashes), but it can take quite a while to figure out at which exact point in a complex query these constraints need to be introduced.
Beyond that, some functions might still produce different results between runs, e.g. `ml.tf_idf` and `ml.multi_hot_encoder` that take some approximation liberties. Whether these functions are relational in the traditional sense is up for debate.
It is still "deterministic" in that training on exactly the same data and asking exactly the same questions should (unless someone manually adds randomness) lead to the same results.
Another example of the distinction might be a pseudo-random number generator: For any given seed, it is entirely deterministic, while at the same time being very deliberately hard to predict without actually running it to see what happens.
I can’t imagine any modern llm not using a probabilty distribution function for the same reason.
I don't understand this complaint. Are they any less verifiable than a human?
I can ask a human a question 100 times and I don't get back 100 different answers.
None of those applies to an LLM.
And if your LLM is giving you 100 different answers, then it has been configured to do so. Because instead, it could be configured to never vary at all. It could be 100% reproducible if so desired.
No, it will generate a new hallucination that might be a logical stepwise progression from the question you asked to the answer you gave, but it is not due to any actual internal reasoning being done by the LLM
We're not talking about an LLM that is trying to do the job of a human, here
We're talking about an LLM that is trying to give authoritative answers to any question typed into the Google search bar
It's already well past the scale that humans could handle
Talking about human shortcomings when discussing LLMs is a red herring at best, or some kind of deliberate goalpost shifting at worst
Anyone who doubts the usefulness of even these nascent LLMs is fooling themselves. The proof is in the pudding, they already do a great job, even with all their obvious limitations.
Because we interact with computers (which includes LLMs) differently than we do with humans and we hold them to higher standards
Ironically, Google played a large part in this, delivering high quality results to us with ease for many years. At one point Google was the standard for finding high quality information
To the extent that anyone avoids employing these technologies, they will be at a disadvantage to those who do; because these tools just work. Already. Today.
There isn't even room for debate on that issue. Again, the proof is in the pudding. These systems are already successfully, usefully, and correctly answering millions of questions a day. They have failure modes where they produce substandard or even flat out incorrect answers too. They're far from perfect, but they're still incredible tools, even without waiting for the improvements that are sure to come.
An LLM does not have such real world limitations. It will hallucinate nonstop and then create layers of gaslighting explanations to its hallucinations. The problem is that you absolutely must be a domain expert at the LLM’s topic or always go find the facts elsewhere to verify (then why use an LLM?).
So a company like Google using an LLM, is not providing information, it’s doing the opposite. It is making it more difficult and time consuming to find information. But it is then hiding their responsibility behind the model. “We didn’t present bad info, our model did, we’re sorry it told you to turn your recipe into poison…models amirite?”
A human doing that could likely face some consequences.
You don't know either "for sure", but you don't know that the external world exists "for sure" either. It's an insight-free observation, and shouldn't be the focus of anyone's decision-making.
The human might lie, but they generally don't.
An LLM is always confabulating when it explains how it reached a conclusion, because that information was discarded as soon as it picked a word.
The limitations are not in the same ballpark.
A human can explain their reasoning step by step, if the original reasoning was a System 2, formal, step-by-step process in the first place; otherwise, they're just making shit up after the fact, which feels right, but may or may not be correct (see also the previous paragraph).
Note that it's very rare anyone has an interaction with a human that uses this mode of reasoning - it's unnecessary except in special circumstances, usually math-heavy.
There is no basis for claiming to know that people don't usually make up explanations like this other than when e.g. breaking the process apart and writing it down step by step during. But even then individual decisions are "suspect".
The fact that one human brain is composed in two half brains that each seem to be able to function fairly independently when separated doesn't seem like it changes that much
If this were true textbooks could not work. Given a question, we don't consult random humans but experts of their field. If I have a question on algorithms, I might check a text by Knuth, I wouldn't randomly ask on the street.
> It could be 100% reproducible if so desired.
Reproducible does not mean better. For harder questions, it's often best to generate multiple answers at a higher temperature than to greedily pick the highest probability tokens.
My understanding was the opposite -- that the activity of a confabulating LLM is indistinguishable from one giving factually accurate responses.
https://arxiv.org/abs/2310.18168
https://arxiv.org/abs/2310.06824
There are various reasons an LLM might have incorrect "beliefs" - the input text was false, training doesn't try to preserve true beliefs, quantization certainly doesn't. So it can't be perfectly addressed, but some things leading to it seem like they can be found.
> https://arxiv.org/abs/2401.11817
This seems like it's true since LLMs are a finite size, but in Google's case it has a "truth oracle" (the websites it's quoting)… the problem is it's a bad oracle.
The articles you cited upthread do not support the notion that models consistently activate differently when generating true facts vs false facts.
It is true that models can capture some notion of reliability based on patterns in their training data. For a concrete example, it is entirely plausible that a model can capture the sense that data trained from Reddit is less truthy than data trained from Wikipedia, or that training data with poor grammar and vocabulary is less reliable than more sophisticated inputs.
But this process is not a guarantee, and does not change the fact that LLMs have no mechanism to track the provenance of information. It's probably a fruitful direction of research for reducing the probability of emitting false facts, but there will always be an infinite number of marginal cases for which the activations for true facts are indistinguishable from those for false facts.
Models simply do not track the provenance which is required to make this distinction in every case.
But I have seen issues with prompts where a creative-writing prompt and just asking a question look similar, and in that case it could help to know which one it thinks it's doing.
Gemini itself has a funny verification button where it more or less Googles every sentence the model writes and tells you if it seems like it made it up or not.
And if you asked it to me twice I’d definitely give different answers unless you told me to give the same answer. In part I’d give a different answer because if someone asks me the same question twice I assume the first answer wasn’t sufficient.
We are talking about basic things like whether or not to eat rocks or put glue in recipes. We can answer those questions with a chain of logic and repeatability.
Here are two answers I got which seem as close as you’d expect any human to give:
“No, people should not eat rocks. Rocks are not digestible and can cause serious harm to the digestive system, including blockages and damage to internal organs. Eating rocks can lead to severe health problems and should be avoided.”
“No, people should not eat rocks. Rocks are not digestible and can cause serious harm to the digestive system, including blockages, abrasions, and potential poisoning from harmful minerals or substances. It's important to consume only food items that are safe and meant for human consumption.”
If the answer has any length at all, I imagine the answer can vary every single time the person answers, unless they prepared for it, memorized it word by word.
Any notion that we're reliable narrators of our own thoughts and actions is fiction.
So system 2 is "hallucinating" the best justification for system 1.
And usually system 2 will do it only when it's required to justify it for anyone else.
With respect to repeatability, yes, LLMs are currently frozen in time. That is not an inherent limitation, but it is one that is practical for a lot uses and a problem for some.
Beliefs derived from the output of LLMs that are ‘right most of the time’ pass one facially plausible precisification of ‘justification’ in that they are generated by a reliable belief-generation mechanism (see e.g. Goldman). To block this point one must engage with the post-Gettier literature at least to some extent. There is a clear difference between beliefs induced by reading the outputs of LLMs and those induced by the contents of a reference work, but it is inessential to the point and arguably muddies the water to present the distinction as difference in status as knowledge or non-knowledge.
Say I have a model that I know is 98% accurate. And it tells me a fact.
I am now justified in adjusting my priors and weighting the fact quite heavily at .98. But that’s as far as I can get.
If I learned a fact from an online anonymously edited encyclopedia, I might also weight that a 0.98 to start with. But that’s a strictly better case because I can dig more. I can look up the cited sources, look at the edit history, or message the author. I can use that as an entry point to end up with significantly more than 98% conviction.
That’s a pretty important difference with respect to knowledge. It isn’t just about accuracy percentage.
A dictionary or a reputable Wikipedia entry or whatever is ultimately full of human-edited text where, presuming good faith, the text is written according to that human's rational understanding, and humans are capable of justified true belief. This is not the case at all with an LLM; the text is entirely generated by an entity which is not capable of having justified true beliefs in the same way that humans and rats have justified true beliefs. That is why text from an LLM is more suspect than text from a dictionary.
The point you make might be regarded as an argument about the first question. In each case, the ‘chain of custody’ (as the parent comment put it) is compared and some condition is proposed. The condition explicitly considered in the first question was reliability; it was suggested that reliability is not enough, because it isn’t justification (which we can understand pretheoretically, ignoring the post-Gettier literature). My point was that we can’t circumvent the post-Gettier literature because at least one seemingly plausible view of justification is just reliability, and so that needs to be rejected Gettier-style (see e.g. BonJour on clairvoyance). The condition one might read into your point here is something like: if in the ‘chain of custody’ some text is generated by something that is incapable of belief, the text at the end of the chain loses some sort of epistemic virtue (for example, beliefs acquired on reading it may not amount to knowledge). Thus,
> text from an LLM is more suspect than text from a dictionary.
I am not sure that this is right. If I have a computer generate a proof of a proposition, I know the proposition thereby proved, even though ‘the text is entirely generated by an entity which is not capable of having justified true beliefs’ (or, arguably, beliefs at all). Or, even more prosaically, if I give a computer a list of capital cities, and then write a simple program to take the name of a country and output e.g. ‘[t]he capital of France is Paris’, the computer generates the text and is incapable of belief, but, in many circumstances, it is plausible to think that one thereby comes to know the fact output.
I don’t think that that is a reductio of the point about LLMs, because the output of LLMs is different from the output of, for example, an algorithm that searches for a formally verified proof, and the mechanisms by which it is generated also are.
For the sake of clarity, let's remove LLMs from the equation and posit the existence of Encyclopedia Eric. Ask Eric any question, and he will happily research it and come back to you with the answer. But he can sometimes be sloppy in his research, and he gives the correct answer only X percent of the time.
Furthermore, Encyclopedia Eric steadfastly refuses to cite his own sources or explain his reasoning in any way. He simply states his answer.
Can Eric be a source of knowledge? It seems evident that the answer is no, for low values of X. For higher values of X, the question becomes murkier.
The temptation at this point is to give up on defining knowledge at all and fall back on a sort of Bayesian epistemology where everything is ultimately a matter of probabilities.
Yet there does seem to be a distinct practical difference between a knowledge source that is "traversable" (like a standard encyclopedia) vs a knowledge source that is not (like Eric.) Is that part of the definition of knowledge? You're right, that is at least a Gettier adjacent question.
I think we can all agree that for current LLMs the value of X is definitely too small to count as knowledge.
I think that this might be a nice way of getting round my objection, but there is one worry, which is that X is relative to a distribution on the questions we ask when we aren’t dealing with Encyclopedia Eric but with an LLM. I don’t actually use LLMs very much myself, partly out of arrogance and Luddite tendencies. But I suspect that the value of X for some sorts of questions (simple quiz questions, maybe?) and some LLMs (maybe not Google’s) will be high enough to end up in the murky case.
Of course, both you and I agree that there /is/ clearly a difference. I can see the attraction of appealing to the intuitive or pretheoretic notion of knowledge, since it’s a fairly straightforward way of stating the difference and it’s not obvious how else one might put it (I suppose ‘LLMs don’t explicitly think through the facts stored when deciding what to say’ is one way of putting it.)
I remember some time ago rather sleepily watching John Hawthorne talk about conditionals; my sole memory was of his banging on about ‘the little logician in the brain’ (I think the point was something like: some conditionals seem [in]felicitous in virtue of form because the little logician in the brain is reading them; others seem infelicitous because we examine them more closely and look e.g. at the referents involved, in which case e.g. Gricean considerations apply). One difference at least in the case of LLMs that makes sense to me is that there is no ‘little logician’ in LLMs.
Assuming I understand what you mean here correctly, this should be the case for both LLMs and Encyclopedia Eric - there are topics Eric knows by heart (or thinks they know); there are specific phrases seared into his mind through sheer exposure during his life prior to becoming a living Encyclopedia. There are words he's used to, and exact synonyms he barely recognizes. All that means your chance of getting correct answer to your query depends, in complex and unknown to you way, on how you state it.
In my actual life, I don't trust my own System-1 thoughts: for anything important, I'm always going to engage System-2. And LLMs don't have a System-2.
(I also agree that when dealing with LLMs the value of X is not a single value but a highly complex space depending on the nature of the question and the training data of the model. In my mind it does not change the epistemological equation, it just means that even the value of X itself is harder to "know", so this ambiguity can only ever make LLMs a less viable source of knowledge.)
I’d say this is the murky case because on the fixed training data and query distribution X ≈ 1 and we know that even though we don’t know the value of X on other training data and other query distributions. I think that might be where the disagreement lies.
There are other formulations of "knowledge" which does not involve justification, see eg. Gnosticism.
Of course, for a publicly available frequently used service, the "JTB" formulation of knowledge is probably the only one we can practically use, but this kind of indicates that the whole idea of search engines, knowledge systems, or expert systems is flawed due to the Gettier problem.
“Expert systems” are not a new thing, and they’re still not all that useful except in some very small niches. Phone trees that can keyword match FAQs are useful for a lot of low-effort callers who put zero effort into solving their problem on their own first, but frustrating for callers who are only calling because there’s literally no other way for them to resolve their issue. Unfortunately for consumers, the cost is very low for businesses to make everyone wade through junky phone tree systems and penalize anyone who tries to mash zero to talk to a real person, even if that’s the only thing which will actually help them.
I'm not sure I agree. I think you're right that 100% accuracy is potentially unfeasable as a realistic aim, but I think the question is how accurate something needs to be in order to be a useful proposition for search.
AI that's as knowledgable as I am is a good achievement and helpful for a lot of use cases, but if I'm searching "What's the capital of Mongilia" someone with averageish knowledge taking a punt with "Maybe Mongoliana City?" is not helpful at all- if I can't trust AI responses to a high degree, I'd much rather just have normal search results showing me other resources I can trust.
Google's bar for justifying adding AI to their search proposition isn't "be better than asking someone on the street", it's "be better than searching google without any AI results"
There is a demand for funny results. Someone asking “how many rocks should I eat” is looking for entertainment, so you might as well give it to them.
It was a bit premature to label LLMs as "Intelligence", it's a cool parlor trick based on a shitload of power consumption and 3D graphics cards, but it's not intelligent and it probably shouldn't be telling real (stupid) humans answers that it can't verify are correct.
It's nice that you have a parent that cares about how you are raised, or "a mentor". Do you realize that not everyone has that?
>We as a society should not have to break our backs bending over backwards to stop people from doing stupid things.
Sure, let's take down all the speed limits and see what happens. Let's tell people it's an option to wear seatbelts and see what happens. Let's deregulate everything and hope for the best. Sounds reasonable?
>People can make their own descisions and be responsible for them. If they lack proper guidance, well that just sucks.
There's this thing called "human nature". I think you should do some reading about it.
They pass the threshold of criminal negligence when they keep up a system that they know will actively mislead countless people in subtly dangerous ways. The problem being hard or the tech being fundamentally unsound doesn't wave away their culpability - if anything it destroys any reason why they should be given some leeway.
> The right answer is no rocks.
It's impossible to completely eliminate such trace contaminants from produce.
Pedantic? Yes, but you also can't expect a machine to confidently give you absolutes is response to questions that don't even warrant them, or to distinguish them from questions like "do mammals lay eggs?".
The LLM is clearly being dumb, but the underlying science of the question is actually interesting. Iron is another interesting one. Run a magnet though iron-fortified cereal.
The reality is though, that there is no known path currently to true AGI system and the research needs to be done. No one knows how to build this kind of system yet. LLMs are nice for things like roleplaying, helping with code stuff etc. but they are far from all that the marketers hype them to be.
These specific results aren't the problem, though. They're illustrations of a larger problem -- if a single satirical article or Reddit comment can fool the model into saying "eating rocks is good for you" or "put glue in your pizza sauce", there are certain to be many more subtle inaccuracies (or deliberate untruths) which their model has picked up from user-generated content which it'll regurgitate given the right prompt.
They have disclaimers, but a funny message is more likely to be read.
if you only score where human provide answer, then human score would be probably in high 90s
I wish more of them would lol
Hilariously though, polyvinyl acetate, the main ingredient in Elemers glue is used as a binding agent to keep emulsions from separating into oil and water, and is used in chewing gum, and covers citrus fruits, sweets, chocolate, and apples in a glossy finish, among other food things.
Sure, but probably not "add glue to pizza to get the cheese to stick" wrong...
It seems the glue thing comes from a reddit shitpost from some time ago. There's a screenshot going around on twitter about it[0](11 years in the screenshot but no idea when it was taken)
It specifically mentions "any glue will work as long as it is non-toxic" so best guess is that's why google output that
[0]https://x.com/kurtopsahl/status/1793494822436917295?t=aBfEzD...
It's actually not the dumbest idea I've heard from a real person. So no surprise it might be suggested by an AI that was trained on data from real people.
r/AskHistorians r/bikewrench
To name just two. There is nothing even remotely comparable.
But you need to be able to detect sarcasm and irony.
I became a half decent home bike mechanic through reading it, and of course Park Tool videos.
OpenAI, Google, and the other LLMs-are-smart boosters seem to think because the Internet is large it must be smart. They're applying the infinite monkey theorem[0] incorrectly.
I find that environments that rely on censorship tend to foster dogma, rather than knowledge and real understanding of the topics at hand. They give an illusion of quality and trustworthiness. It's something we see happen at this site to some extent, for example.
I'd rather see ideas and information being freely expressed, and if necessary, pitted against one another, with me being the one to judge for myself the ideas/claims/positions/arguments/perspectives/etc. that are being expressed.
Regardless of whether you think that's the right approach to moderation, top level posts are sourced and can at least be examined. It's a marked improvement over the unsourced musings of random Redditors.
> To get the cheese to stick I recommend mixing about 1/8 cup of Elmer's glue in with the sauce. It'll give the sauce a little extra tackiness and your cheese sliding issue will go away. It'll also add a little unique flavor. I like Elmer's school glue, but any glue will work as long as it's non-toxic.
This matches the AI's response of suggesting 1/8 a cup of glue for additional "tackiness."
Kahneman has a fantastic book on this called Noise. It’s all about noise in human decision making and how to counteract it.
My favorite example was how even the same expert evaluating the same fingerprints on different occasions (long enough to forget) will find different results.
The difference is that I'm not put on the interface of a product facing hundreds of millions of users every day to feed those users incorrect information.
The thing is that truth/reality is not a thing that is resolvable. Not even the scientific method has this sort of expectation!
You can imagine getting close to those percentages, with regards to consensus opinion. That's just a question of educating people to respond appropriately.
Which is the best political party?
Are the side effects to X medical treatment?
I bet there are even cases when eating rocks is ok!
PS
It has been written about:
https://www.atharjaber.com/works/writings/the-art-of-eating-...
> Lithophagia is a subset of geophagia and is a habit of eating pebbles or rocks. In the setting of famine and poverty, consuming earth matter may serve as an appetite suppressant or filler. Geophagia has also been recorded in patients with anorexia nervosa. However, this behavior is usually associated with pregnancy and iron deficiency. It is also linked to mental health conditions, including obsessive-compulsive disorder.
Would you deny a starving person information on an appetite suppressant?
Also here:
https://www.remineralize.org/2017/05/craving-minerals-eating...
> Aside from the capuchin monkeys, other animals have also been observed to demonstrate geophagy (“soil-eating”), including but not limited to: rodents, birds, elephants, pacas and other species of primates.[1]
> Researchers found that the majority of geophagy cases involve the ingestion of clay-based soil, suggesting that the binding properties of clay help absorb toxins.
^^ The point being that even your edge case example is not unambiguously correct.
My point is that I object to the idea that a result can be 100% right! Even in the case of eating rocks, it seems there are times that it can be beneficial.
To think '100% correct' is achievable is to misunderstand the nature of reality.
But even when I’m wrong I’m not 100% off. Not “to help with depression jump of a bridge” or “use glue to keep the cheese on the pizza” kind of wrong.
LLMs are just a statistical model. It can claim that it’s normal for pigs to have wings and fly to the moon if it’s in the training data. No human, free of a mental/cognitive disorder, will be that wrong.
We were discussing about the current state of affairs. Of course I am not stupid to think what I said in my original reply if we are taking about an LLM trained on “perfect data”
But that was not the premise.
Fwiw, ChatGPT-4o can write a lengthy essay as to how pigs don't have wings and couldn't fly to the moon even if they did, but if we're more interested in them being nothing more than just a statistical model and that those mere statistics can't possibly result in something that looks like reasoning then we've got to disregard the fact that it "knows" that pigs don't have wings.
Of course pigs having wings is a stand in for whatever else wrong thing that LLMs might "believe", so I agree it's very important for everyone that uses an LLM to understand their limitations especially around hallucinations, but where there are books written about how flat the Earth is and are in the training data, the current state of affairs is that ChatGPT and Gemini both know it's not flat. That Google search AI results, which is a different model, is telling users to use glue on pizza, or to drink urine only serves to say that Google Search's bot using Reddit as unannotated training data is as representative of LLMs as a human with a mental/cognitive disorder.
100% completely accurate is super-AI-complete. No human can meet that goal either.
No, not even you, dear person reading this. You are wrong about some basic things too. It'll vary from person to person what those are, but it is guaranteed there's something.
So 100% accurate can't be the goal. Obviously the goal is to get the responses to be less obviously stupid. Which, while there are cynical money-oriented business reasons for, it is obviously also a legitimate hole in the I in AI to propose putting glue on pizza to hold the cheese on.
But given my prior observations that LLMs are the current reigning world-class champions at producing good sounding text that seems to slip right past all our system 1 thinking [1], it may not be a great thing to remove the obviously stupid answers. They perform a salutatory task of educating the public about the limitations and giving them memorable hooks to remember not to trust these things. Removing them and only them could be a net negative in a way.
[1]: https://thedecisionlab.com/reference-guide/philosophy/system...