anthropomorphizing?

I think this is a great point thats not being acknowledged. There's a distinction in psychology between intellectual understanding and insight. Insight involves "the expansion of the ego by self-observation, memory recovery, cognitive participation, and reconstruction in the context of affective reliving". i.e.: Realising on a deep emotional level that something is true and integrating that into our core understanding of the world.

In this case we have a dissonance between an intellectual understanding that these models are in some sense just predicting where the conversation should go, and our false insight that they are persons. The illusion of emotion and engagement is strong and getting stronger.

It starts with folks like Blake Lemoine, but ultimately we're all vulnerable. Just as we're vulnerable to increasingly tailored and sophisticated forms of media manipulation. An understanding of the mechanics of media narratives, broadcasting etc is not a strong enough defence.

The 'insight fallacy' is the idea that understanding the nature of a problem will be enough to solve it. In this case we have a whole bunch of people studying alignment, and attempting to understand the kind of interior models LLMs form. While simultaneously improving them and commodifying them.

It goes beyond a perverse profit incentive. As a species we have a deep desire for non-human companionship - be it in the form of spirits, aliens, A.I. or even pets. We seem determined to create it, and wildly emotionally vulnerable to it in even the most primitive forms. From ELIZA on down, when people have engaged with conversational agents, they have overridden our better judgement. False insight and false friendship, is a major social vulnerability.