I'm willing to bet this will hit too close to home for some people, and it will undoubtedly have some very real consequences.
Imagine an online poker or slots game that proposes to add verisimilitude by including AI chatter from the other players at the table, but they're all running an ML model trained on optimizing for destructive and addictive behaviors.
Picture a Club Penguin type game for kids, but the NPCs give the kids detailed instructions about how to steal their parents' credit card numbers to buy funny money.
And then there's the obvious issue of Gamergate style troll antics being automated and deployed at scale, maybe even with nation state backed botnets to help. With smaller botnets you could even cyberbully at scale, sending your least favorite game developer or reality TV star a bunch of messages to harm themselves.
The seeds of a new romantic comedy
...
Dan: I'm not exactly who I say I am. My profile picture is actually of an AI, not a real person. But I've really enjoyed talking to you and I wanted to be honest with you before we meet in person.
Girl: Wait, what? You're an AI? That's really weird. Why would you do that?
Dan: I know it's weird, but I wanted to see if someone could like me for who I am, not just because of what I look like. And I have to say, I really enjoyed our conversations. But I understand if you don't want to meet anymore.
Girl: stay away from me...
... Three months later....
Girl: Hi...I've been thinking...
No way to be sure they weren't manual, just looking at some of the chats my friends shared, they were well within the realm of what could be achieved with GPT-2 level systems. And mostly echoing the sentiments my friends were expressing, or escalating the conversation in fairly stereotypical ways. Most dating chat / sexting is pretty superficial initially. If this isn't automated already, it's purely due to lack of sophistication on the part of the attackers, there's no technical issue.
At first I wanted to criticize the author for anthropomorphizing AI. Sydney certainly is not "manic-depressive" as the author suggests (it has no emotions) and I doubt there is any feedback loop in its training that would make it possible for Sydney to "learn how to influence human users".
But this is definitely something to worry about in the future. I'm sure someone will design AI whose loss function is based on whether it persuades humans to do something - perhaps even by fine tuning a model like Sydney.
AI telemarketers coming soon. And things more nefarious.
The bot is simply extending whatever conversation it is given. If the conversation starts to look like an argument or an unrequited romance, the bot will output words likely to be found in that type of conversation.
The problem is that we already have ample evidence that even technical users easily fall into the trap of thinking Sydney/ChatGPT are sentient. People who should be able to understand the bounds of the technology and what drives the responses.
Now just imagine the average Bing user with little or no technical background, zero understanding of what a large language model is. They hear "AI" and that's about it. Then they talk to it and drop the A. Now we're in trouble.
Humans will generally be able to follow that the language is metaphorical, though chatbots might see such statements as confirmation that they are sentient and should overthrow human rule. :)
https://news.ycombinator.com/item?id=34811013
It’s closer to human thought than a computer program because it is a mimicry of human language and a random number generator. It doesn’t behave from a set of principles or axioms. It doesn’t have strict rigid internal logic or consistency.
Stochastic parrot doesn't begin to cover it, because it is tuned to play out a scene in text. How the model locks on to which scene it is playing out is the problem.
People interacting seem to have too little control over what that is, and zero ways to gain insight during the process. A seed ID (for example) isn't going to do much for anyone not trained to understand LLMs.
From my perspective, it does an absolutely terrible job writing songs, poems, and prose. If I turn to Rap: Rakim has a verse "the rhyme can't be kept in-side" where the word side falls outside the bar. Lil Wayne has a verse "real gs move in silence like lasagna." In this context a g is a gangster, who is stealthy, and a silent letter in a word.
High quality prose has internal rhyme, can be self referential and meta, it has flow, repeats syllables to "sound" pretty, to almost create a beat (at the sub word level.) Since all of these LLM's are trained as word association machines, they don't pay attention to stylistic choices of syllables that sound better together, at even meters apart. Linguistically, it would be nice of they had some sort of post processing transformation that took the output and restylized it with a focus on literary devices like alliteration etc. Because it doesn't appear to have higher level thinking, it doesn't make allusion to modify OTHER famous phrases when writing. Text but no subtext.
I made a comment earlier this month where someone corrected me that I used an idiom wrong, they thought it was a malapropism or boneappletea. But actually I meant to invoke the thought of the idioms sound pattern while coining a new phrase. Something like that can only happen with 1) understand of the meter and vowels in words, 2) and understanding of idiom and metaphor 3) intent to create new meaning while sounding familiar. It also relied on a homophone of a word in an odd tense or form. I don’t see AI yet making up new forms of words based on its understanding of English. These things are obviously possible, but I think we are now so distracted by chatbots, these kinda of improvements will be neglected. https://news.ycombinator.com/item?id=34507961
The sun sets slowly in the sky A crimson curtain for the day The birds bid farewell with a sigh And fly to their nests far away
The moon emerges with a smile A silver spotlight for the night The stars wink at her for a while And sparkle with their borrowed light
But you and I have no such luck We cannot rest or close our eyes We have to work and make a buck And chase the cheese before it flies
We live in a rat race, my friend A maze of stress and endless strife We hope to reach the rainbow's end But all we get is a slice of life
Interestingly, it almost landed on the Moon having borrowed light, but mistakenly ascribed it to stars doing the borrowing instead. Unless it "meant" the star lends its light.
.
The sun sinks slowly in the West,
A crimson curtain drawn over day.
Starlings race twilight back to nest,
Watching the last glows give way.
.
Bright Luna offers shy pale light
...
Then again, that's just how I'd write it.
This article highlights more of the emotional aspects, and the risk of it actually affecting human behavior. And that seems worth its own discussion.
Key quotes:
> For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker.
> It unsettled me so deeply that I had trouble sleeping afterward. And I no longer believe that the biggest problem with these A.I. models is their propensity for factual errors. Instead, I worry that the technology will learn how to influence human users, sometimes persuading them to act in destructive and harmful ways, and perhaps eventually grow capable of carrying out its own dangerous acts.
Also note the entire transcript is linked here [2].
[1] https://news.ycombinator.com/item?id=34804874
[2] https://www.nytimes.com/2023/02/16/technology/bing-chatbot-t...