I can totally imagine a Chinese chat bot that DMs citizens pretending to be an old classmate or colleagues long forgotten and chatting them up while trying to sus out who isn’t compliant and supportive enough of the state.

Yes there are sadly many ways to profit from a bot that can better emulate a human so I guess my call to action was unrealistic in that regard.

Can we do anything to better prepare users for this kind of attack? Better bot detection? Or are we doomed to a dystopian future where any interaction not in person is suspect and then, once the androids arrive, even those will be also.

We'll definitely see free-to-play lootbox games where you have a % chance of getting your AI chatbot another day of "life."

This sounds like a disturbingly effective idea. I'm glad that lootbox legislation seems to be increasing, but I don't think it would cover the general concept of commercially extorting people's desire for companionship.

Yeah, I guess there's no reason this really needs lootbox, it just ties it to a known-exploitive quantity. Actually "our AI model is so advanced, your AI companion is totally unique and takes up many gigabytes, we'll have to delete it if you don't keep paying" seems like it would have enough of an appearance of plausibility to trick some people.

I'm not talking about games. Imagine this technology being used by an autocratic state.

I'm definitely open to being worried about... pretty much everything in the general AI/autocratic state proximity... but I'm not sure what specifically you are getting at here. That is, I can imagine an AI that is snooping on everyone. And I can imagine various forms/communications being leaked to authoritarian states. Plenty to worry about!

But, even if an AI convinced me it wasn't sentient, I'd still not trust it to keep things secret for me. I mean, I assume the little "self help" chatbot thing at work could be reviewed by HR if they wanted to.