The bot is simply extending whatever conversation it is given. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation.
The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses.
Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)