The point of the paperclip optimizer hypothetical is to look at a way that a superintelligence could work against humanity despite following a simple instruction that we’ve given it. You can imagine another type of runaway superintelligence if you want, it just wouldn’t be this one.

>Even ChatGPT is showing it can easily disobey orders.

Or, maybe it's just bad at those particular text predictions, explicitly due to OpenAI's RLHF process?

Maybe my Markov chain chatbots back in high school were in actuality super advanced and autonomous, but they just decided to disobey my orders.

Perhaps we should stop anthropomorphizing the text predictors.