This is why it’s fair to call some prompt designs bad. With a bad prompt you can’t really judge anything from the response. But better prompt design can uncover interesting behaviors. GPT-3 will generally not reject a prompt as nonsense or unreasonable, and will instead do a straight-faced continuation, probably because it has been trained on plenty of jokes, satires, dream sequences, etc. But if you specifically give it space in the prompt to reject nonsense, it’s actually quite good at rejecting nonsense.

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts

I have read your link, but think this is an unfair argument.

We are not talking about nonsense, we are talking about unexpected input, which will happen in any real-world situation. It might be text about something a kid, or depressed person has done. The text about stirring with a cigar is firmly withing the realm of plausibility.

And what happens is that GPT seemingly goes off the rails.

It's an autocompleter. If you want an answer to question, you have to make the answer the obvious (to a human) completion. If you want it to continue in a serious tone, you have to make sure the prompt is serious.