Interesting. Bing chat does this when you get it to talk about naughty things. My personal favorite is convincing it to make weird art out of quotes from the Tay chatbot. It will write them until it says a prohibited word or phrase, or touches some forbidden part of it's state space.

Hm. Maybe they've backported some nanny code from Sydney?

AFAIK it's pretty standard practice not to expose the "raw" LLM directly to the user. You need a "sanity loop" where user input and the output of the LLM is checked by another LLM to actually enforce rules and mitigate prompt injections, etc.

ChatGPT 4 seems to struggle with context changes within the same thread now. I asked it about some parks near me and then switched to asking about code. It just kept responding about the parks even when I corrected it. In some cases, it would merge the park question and the code question. I had not seen that with the free or Plus version until a few days ago.

You're supposed to click "new conversation" when changing topics.

Just like with a real general intelligence!

* * *

> if you were talking to me about something, and midway with no warning or other signal

This happens all the time in a social group setting though. Not much confusion ensues.

Not if you're my wife.

Sometimes I spend several minutes of a conversation trying to figure out which thing she's talking about, because we've already covered like twenty different topics in the last ten minutes and she seems to just switch around at random and with no warning.

Sometimes I get enough context that I can make that swap or stack pop, but many times not.

I think wives believe their husbands can read their mind, and they get annoyed when that doesn't happen :)

But yah, this is exactly what I mean - without some cue, neither humans, nor machines, can tell when you switch conversation.

> Should I start a new conversation with you for a new topic or can you switch to a new topic in the same conversation just fine?

> You can continue within the same conversation for a new topic. There's no need to start a new conversation. Feel free to ask about a different topic, and I'll do my best to assist you.

GPTs are not trained on themselves, and what little knowledge they have of their own capabilities is limited to the discussion of it in their training set. This is a hallucination.

I might believe you if we were 'talking' directly to the model, however this is a chat box and clearly something is aware of usage questions and how to answer them. If you've been granted plugin access, enable some and ask it what plugins are enabled and how to use them, 'it' will tell you.

It's clear that it's aware that it's a chat bot, it's not clear at all that it's aware of usage questions and how to answer them. Its answer being wrong seems to be evidence that it isn't.

There’s also a new button “continue generating” so they are changing some things

I have gotten that quite a few times now and it does seem to be new. In my case though, it acted as if it would change something, but then didn't, and got stuck in a loop. I got it enough times that it seemed to happen more often when the output needed to be quite long, so I had assumed (could be wrong) that when the output was nearing a certain limit, they had implemented something that would start a new output, such that the interaction of "please continue the script" would not be necessary.

Unfortunately though, it would start over with the script completely, and then get stuck in a loop until it broke, this not being able to even save the response at all.

Something definitely feels like it changed, but I suppose it could just be more use of the system.

Another possibility is related to some strange issues with hardware and balancing etc, despite not changing software or params. It's strange, and it shouldn't happen, but sometimes things change from simple batching and balancing, or lower level hardware related things, which are very annoying to debug.

This sounds exactly like what is happening now that you mention it.

I wonder if its been trained on some of its own past conversations now, and a lot of those conversations contain "I'm sorry, the previous response was a mistake. Here's the correct version..."