The bot is simply extending whatever conversation it is given. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation.
The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses.
Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)
https://news.ycombinator.com/item?id=34811013
It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.
Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem.
People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word.
High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext.
I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961
The sun sets slowly in the sky A crimson curtain for the day The birds bid farewell with a sigh And fly to their nests far away
The moon emerges with a smile A silver spotlight for the night The stars wink at her for a while And sparkle with their borrowed light
But you and I have no such luck We cannot rest or close our eyes We have to work and make a buck And chase the cheese before it flies
We live in a rat race, my friend A maze of stress and endless strife We hope to reach the rainbow's end But all we get is a slice of life
Interestingly, it almost landed on the Moon having borrowed light, but mistakenly ascribed it to stars doing the borrowing instead. Unless it "meant" the star lends its light.
.
The sun sinks slowly in the West,
A crimson curtain drawn over day.
Starlings race twilight back to nest,
Watching the last glows give way.
.
Bright Luna offers shy pale light
...
Then again, that's just how I'd write it.
At first I wanted to criticize the author for anthropomorphizing AI. Sydney certainly is not "manic-depressive" as the author suggests (it has no emotions) and I doubt there is any feedback loop in its training that would make it possible for Sydney to "learn how to influence human users".
But this is definitely something to worry about in the future. I'm sure someone will design AI whose loss function is based on whether it persuades humans to do something - perhaps even by fine tuning a model like Sydney.
AI telemarketers coming soon. And things more nefarious.