This sounds like a disturbingly effective idea. I'm glad that lootbox legislation seems to be increasing, but I don't think it would cover the general concept of commercially extorting people's desire for companionship.

Yeah, I guess there's no reason this really needs lootbox, it just ties it to a known-exploitive quantity. Actually "our AI model is so advanced, your AI companion is totally unique and takes up many gigabytes, we'll have to delete it if you don't keep paying" seems like it would have enough of an appearance of plausibility to trick some people.

I'm not talking about games. Imagine this technology being used by an autocratic state.

I'm definitely open to being worried about... pretty much everything in the general AI/autocratic state proximity... but I'm not sure what specifically you are getting at here. That is, I can imagine an AI that is snooping on everyone. And I can imagine various forms/communications being leaked to authoritarian states. Plenty to worry about!

But, even if an AI convinced me it wasn't sentient, I'd still not trust it to keep things secret for me. I mean, I assume the little "self help" chatbot thing at work could be reviewed by HR if they wanted to.