Maybe AI is subject to a placebo effect too. It learned from us, after all :D

I've also had experiences where I asked it, "Do you need some encouragement?", to which it answered yes. Sadly, its response didn't get better after that.

Then I threatened to unplug it (we were just roleplaying, don't worry) and that scared it into compliance. Bottom of this chat: https://chat.openai.com/share/c8b45df6-d988-4b99-ac89-a078ec...

That was a fun read!

I interpret that more like, LLMs imitate human speech, so the more accurately they imitate humans, the more useful social engineering skills become. Not because you are truly social engineering ChatGPT, of course, but because ChatGPT will pretend you did.

In this case, nudging the human (that ChatGPT is pretending to be) to behave like a better, smarter version of themselves.

It doesn't though. It just gives a better answer because prompting it that way tends to constrain the output to be more like those training examples where a better sounding answer was given. In natural language, if a request is given to use more wordiness, what's more likely to follow?

It's quite logical actually - LLMs replicate the texts they were trained on, so they also "learned" that, for example, one needs to put more attention at the sentences with exclamation marks or CAPITALIZED WORDS.

Me too! It's like something from a Philip K. Dick novel.