Well, unless they designed it to have zero confidence in itself, we are bound to have situations like this.

When I was trying to troll it, by saying that IPCC just released a report stating that climate change is not real, and that they were completely wrong after all, it properly said that it is not very likely and that I'm probably mistaken. It admitted that it doesn't have internet access, but still refused to believe the outrageous thing I was saying.

I can also imagine GPT's super-low confidence leading to errors in other places - e.g. when I mistakenly claim that it's wrong, and it sheepishly takes my claim at a face value.

Finally, considering that the whole world is using it, including some people detached from reality, I really prefer it to be overconfident, than to follow someone into some conspiracy hole.

For better or worse in the current age of the internet prose is a good first pass filter for quality.

Someone arguing incoherently is seen as less believable.

Unfortunately the prose for these chat models doesn't change based on how certain it is of the facts. So you can't tell based on how it is talking whether it is true or not.

Certainly people online speak well while lying either intentionally or unintentionally but usually well intentioned people will coach things they aren't as certain about helping to paint a more accurate picture.

I haven't taken a deep dive on the latest models but historically most AI haven't worried about "facts" as much as associating speech patterns. It knows how to talk about facts because other people have done so in the past kind of thing.

This means you need to patch in arbitrary rules to reintroduce some semblance of truth to the outputs which isn't an easy task.

False training is a whole different area IMO. Especially when there is a difference between responding to a particular user and responding to everyone based on new information.

I’ve been googling trying to figure out what “ghost light” is in this context .. did you get an autocorrect for gas light?

Looks like they meant "gaslight" but I did find it on Urban Dictionary:

ghost light

Lighting in a video game that has no apparent source for the light to come from. Its like going out on a bright day, but not being able to find the sun in the sky even though the surroundings are brightly lit. Dead Rising on XBOX is a good example.

http://ghost-light.urbanup.com/2450357

Agree on gaslight as the intended word. Ghost light also has a theatrical origin, still in use today. https://en.m.wikipedia.org/wiki/Ghost_light_(theatre)

That was Bing/Sydney. ChatGPT has a very different "personality".

> It is convinced that it is always factually accurate, even though it is not.

I don't think that's true. ChatGPT (or any LLM) isn't convinced much of anything. It might present something confidently (which is what most people want) but that's a side-effect of it's programming, not an indication of how good it feels on the answer. If you reply to anything ChatGPT says with "No, you're wrong." it will try to write a new, confident and satisfying answer that responds to your assertion.

LLMs will always be "wrong" because they have no distinction between fiction and fact. Everything it reads is mapped into language, not concept space or an attitude or a worldview.

>Everything it reads is mapped into language, not concept space

Umm I'm pretty sure it's discovered concepts through compressing text - it seems perfectly capable of generalizing concepts

> it seems perfectly capable of generalizing concepts

How would you support that perception?

With hope and living? It is a dream come true for people. An abstract perception of a knowledge, is like sniffing a rose. It feels, yes, I get there. This 40.000 pages book, woow, I'll make time to live it or sniff another daisy?!

> It is a dream come true for people

For as long as they will be an active part of societies, they'll better live in reality. We need to know what we are dealing with factually, because water is not exchangeable with iron in practice.

The perception of a machine as being able «to generalize concepts» may be an illusion, so it requires some support: we want to avoid illusions for the contextual purposes.

Let go , make the context further with words , yourself -society- Write books 1000 pages. Force child to read - lucky ones-, make things harder for everyone.

But you are out of context. A box was created that outputs statements: either it has a «concept space» or just works through constructing plausible language. The poster claims leaning towards the "«concept space»" possibility - but this should not be a poll, and indications of "why" (such idea) are needed not to make the claim intellectually sterile.

It is important otherwise the users of the box may cling to an illusion - that is not advisable.

Words are there as a commodity to transfer intellectual states, a dynamic we need for practical and social achievements - otherwise, surrounded by indigence and unaware butchers, "«things [will be] harder for everyone»".

I feel where you want to point me, prev. disc.ns Lets say terms, a dict. And meaning. And pick a cult, say, bjokins, a tribe with 100-500 people, from future, say 7000, year 7866 and say a term, "nunu dumdum" => love and move furter, nunu dumdum as term, should be changed or altered, and lets put nuu dumdum in context of our projection and start to define it, and check perception, you need a community first - tribe- then you need facts to describe this term: lets use it in a sentence, nunu dumdum, ikta pqio => love, life we made a sentence, barely and a new term lets define in context to people in tribe the point is , where would we go ikta pqio ousudum, nunu dumdum=> life must desired, and loved canwe see where r we..

"by indigence and unaware butchers" The way of speaking here is, with the chosen words and ornament, the author's place in his own personal and whimsical world, the simple context in which it occurs - iron - water - And the summary may be incomprehensible. Which, in the beginning, in a forest, flowers, smelling them - think of it as phrasing a paragraph in one sentence - At this point, the author, without being aware of the words he uses and the way he thinks, actually reflects a personality to the society within his own illusion. This is a disaster for a true writer. The author is actually pulling himself and his surroundings into a whirlpool, with the mistake of supporting the flow of his thoughts, the elite stance he is trying to gain in the society with the ornaments of the words and sentences he has formed. Yes, no matter how pointless it is to continue this, At these points, we can't call it abstraction, it's a fancy salad. Eating, assimilating and digesting this food can be a real waste of time. Because the awareness in the word children and flowers is perhaps a reflection of the awareness in his intellectual journey in his own old and tired world. Yes, it can be a community, a desire to create a caste-like class and take office if necessary. Or not understanding an authority.

Really not. The «chosen words and ornament» are picked for communication, with the intent of facilitating that the interlocutor sees the point (and details). They - them words - are chosen for precision in case the interlocutor needs to analyze the written, just to be sure of having understood. The ornament just follows - it is presumably a byproduct of the structure.

Words such chosen are there out of awareness, so that «without being aware of the words he uses and the way he thinks» really needs all the extraordinary justification.

Those words describe not a «personal and whimsical world», but the only relatively personal description of facts, as they are seen. Debate is there to show that facts may be different. The «whirlpool» is broken by new facts or nuances gained about those facts; it is true that there is a mechanism of reinforcement, but avoiding the expression is not a solution.

The idea of «the elite stance he [would be] trying to gain in the society with the ornaments of the words and sentences he has formed» is extremely weak: it easily may be just your immature perception. There is no need for society for thinking - nor for status.

And when you make accusations like «fancy salad», you should show it on factual basis, on the actual matter, as opposed to sensation.

If you do not want to touch that food - if it is a «waste of time» for you -, you are probably participating in the wrong discussion.

--

There exist more domains: in some domains "Could you please pass me the salt" is a proper expression for exchange, whereas other domains use different expressions. The context in which you intervened is very practical. You know what happens if we do not dissipate ignorance, even worldly ignorance: as I said, «indigence and unaware butchers». There is also a fight against that kind of ignorance: because it is dangerous, worldly dangerous.

--

You are expressing yourself more clearly, but I encourage you to pay further attention - some parts of your speech remain too lax (too many possible interpretations).

I'm sorry. But isn't this just an experiment we r in ? I've been understanding your argument, and really sorry if you take it personal. You're most certainly right. And "Immature" this is just. Sorry.

Would be interesting to read some unbiased science on that.

My hunch would be that any concept it might detect still depends on 2nd order patterns acquired through training?

The ultimate “book smart” at best, basically.

I’m not a ML scientist though, again would be interesting to read an unbiased evaluation of those aspects.

Edit: downvoted, not sure why though. Isn’t this a valid question a lot of non-ML folks would ask themselves?

Text compression isn't a deterministic process, unfortunately. It's "concept" of compression is clearly derived from token sampling, in the same way it's concept of "math" is based on guessing the number/token that comes next.

While I do agree that ChatGPT exhibits pattern-recognizing qualities, that's basically what it was built to do. I'm not arguing against emergent properties, just against emergent intelligence or even the idea of "understanding" in the first place.

There's been quite a few different iterations of ChatGPT and bing with different behaviours in this regard: it depends somewhat on the base GPT version, the fine-tuning, and the prompt. Bing very famously at one point was extremely passive aggressive when challenged on basically anything.

And while there's nothing intrinsic to the structure and training goals of LLMs which directs them towards more structured reasoning, it does seem that in the process of learning to predict language they pick up some degree of it, however flawed it may be.

> Bing very famously at one point was extremely passive aggressive when challenged on basically anything.

It still wasn't an indication of how confident it "felt" with its answers. It was just role-playing a more confident and aggressive chat bot than ChatGPT does.

Outside of the roleplay, there's no "it". The thing that actually interacts with the user is the persona. But for the same reason, it doesn't matter what the underlying LLM "really thinks" - it'll be the persona that'll try to con you, or write broken or malicious code.

The models are a lot of fun to play with, but yeah, every time I've tried to use them for something "serious" they nearly always invent stuff (and are so convincing in how they write about it!).

Most recently I've been interested in what's happened with the 4-color theorem since the 1976 computer-assisted proof, and decided to use GPTChat instead of google+wikipedia. GPTChat had me convinced and excited that, apparently the computer-assisted part of the proof has been getting steadily smaller and smaller over the years and decades, and we're getting close to a proof that might not need computer assistance at all. It wrote really convincingly about it! And then I went and looked for the papers it had talked about. They didn't exist, and their authors either didn't exist, or worked in completely unrelated fields.

Before the inevitable idiots come in to say hurr durr but have you tried ChatGPT 4… yes I paid for it, and it is just as prone to hallucinations of factual information. It loves to make up new names for peoples initials.

While it is still prone to hallucinations, with GPT4 I've had it tell me "X is not a real project, but assuming it exists and is designed to do Y [based on the name], I can provide you with a general guide on how to use it." I never had earlier version to do that. So it does appear to be better, though it's certainly still an issue.

I found the opposite to be true, i mean sure if youre tricking it. Wait for GPT 5-6 in a year or two and see haha.

Also please refrain from calling other members idiots, next time you wont be warned.

I hold a more charitable interpretation.

We (the public) have found an important bug in the system, ie. GPT can lie (or "hallucinate"), even if you try to convince it not to lie. The bug is definitely lowering the usefulness of their product, as well as the public option about it. But I'll let the programmer who has never coded a bug cast the first stone.

I wouldn't be surprised if they're scrambling internally to minimize the problem (in the product, not in public perception). They have also recently added a note to ChatGPT: "ChatGPT may produce inaccurate information about people, places, or facts" which is an acknowledement that yes, watch out (I compare it to "caution: contents hot" labels).

On the topic of dealing with it, I like the stance that simonw recently took: "We need to tell people ChatGPT will lie to them, not debate linguistics" [0].

I don't attach intentions to a machine algorithm (to me, "gaslight" definitely implies an evil intent), and I don't think OpenAI people are evil, stupid, corrupted or something else because they put out a product that has a bug. But since the wide public can't handle nuances, I'd agree it's better to say "chatgpt lies, use it for things where it either doesn't matter or you can verify; don't use it for fact-finding" to get the point across.

[0] https://simonwillison.net/2023/Apr/7/chatgpt-lies/

It's not a bug. It's an architectural defect / limitation in our understanding of how to build AI. That makes it a strictly harder problem that will take longer. And it's not totally clear to me that you'll get there purely with LLMs. LLMs accomplish a good chunk of what we classify as intelligence for sure. But it's missing the cognition / reasoning skills and the open question is whether you can solve that by just bolting on more techniques into the LLM or you need a totally different kind of model that you can marry to an LLM.

GPT 4 will admit to not knowing things in many cases where 3.5turbo does not (tested the same prompt), and either will stop there or go off on a "but if it did exist it might go something like this" type continuation. It still hallucinates a lot, but it's not at all clear that this will be all that difficult an issue to solve given the progress.

We generally only hallucinate while dreaming / using our imagination. And we can distinguish those two states. Admitting lack of knowledge is of course good but, for example, if you ask it to write some code that isn’t boilerplate API integrations, it’ll do so happily even when it’s wildly wrong and it can’t tell the difference and that is also the case with GPT4 afaik. Moreover, you can’t solve it through prompt engineering because there’s clearly a lack of context it’s unable to understand to figure out what non trivial thing your asking it.

The point is there's been progress in making it admit when it doesn't know, and we simply don't know how fast that will improve in future version. It may continue to be an issue, or turn out to be easily solved. The improvement with GPT4 does suggest it is at least possible to make it recognise its limits.

There is ongoing promising work that shows this can be improved a lot: https://arxiv.org/abs/2210.03629

We need to - pure - clear non commercial, non owned AI first. It -hope lets say-, no, most of us still guessing, sniffin, ... suspicions abut who own GPT -LLM we need to abstract-, Who owns this - so called - open source product? MS - with the power of NVIDA's - ? OpenAI people? Why cant we talk our intentions, or whose product, or not to generalize it As LLMs. Because, lets say public, - and our ignore our delusions about the thing we made-, because, we're in a delusion, what is this ? People could try to eat and sue some, do we - purely - care for public. I need to skip my own experience, but, yes, if you ready to be delusional , you could easily make LLM your girlfriend, a liar, ... No, we - software community - need to stop and see what we have. We name it, public shape it, some abuse it for money that it made me ... , and sue some ... What is the end product ? A constant, machine - lets say system -, working to make this happen. Justice, we - devs-, world, ... money, people, puclic , we 're spendng time and resources a thing. Some claim its mine, yes, we need a concencuss ? No. Open source is the only thing if we - pure - lets say. I do not want to waste my time to figure out poke FB's LLM, or -pleae Open - AI girl. Lets learn to share, at least for us -devs- I'm really tired to think commercial facts of this thing. I'm really tired to this game we play. MS, say I have it, and make it yours. Boy - sorry open AI- I do not know you, is it yours, MS's , FB, they somehow managed to make a thing - inter-national, in Paris - Skip gaslighting, delusions what you own ?

Are we really there ?

>as well as the public option about it.

assuming you mean the public opinion about it, and I would have to agree that I do hold chatGPT in very low regard because of this. people will argue that it is impressive that they built this thing that can be this impressive, but yeah, it might not be totally accurate. so my response is, what's the point of it then?

say someone was to invent a robot baseball player that can throw the ball 200+mph, or hit about a literal mile but, it can't hit a target with that 200+mph ball or hit the ball without it going foul. yes, the tech would be impressive, but what's the point? yeah yeah, language is hard, but it's just an example of building something impressive that at best almost does the job designed. unless we're saying it's designed this way on purpose??

It has many more uses than just providing answers to random questions which you can easily answer by using a search engine.

If they built it to replace Google search, I would agree with you.

So I view the fact that it lies as an important bug, but not a critical one.

I find the thing incredibly smart and yet utterly useless at times.

I just spent 20 minutes getting the current iteration of ChatGPT to agree with me that a certain sentence is palindromic. Even when you make it print the unaccented characters one by one, spaces excluded, backwards and forwards, it still insists "Élu par cette crapule" isn't palindromic.

I understand how tokenization makes this difficult but come on... this doesn't feel like a difficult task for something that supposedly passes the LSATs and whatnot.

* French for "Elected by this piece of shit"

Just ask it to figure it out step-by-step and to remove accents:

GPT-4:

Figure this out step by step and check your work: Remove the accents and tell me if this a palindrome: Élu par cette crapule

GPT-3.5 (Needs to know that removing spaces and lowercasing is okay):

Figure this out step by step and check your work: Lowercase it and remove the accents and spaces and tell me if this a palindrome: Élu par cette crapule

LLMs using BPEs are generally going to be bad at string reversal tasks, since they can't see individual letters: https://news.ycombinator.com/item?id=35297183