It's called profiling and the NSA has been doing it for at least decades.
Otherwise all they have is primitive swipe gestures of endless TikTok brain rot feeds.
We are already seeing AI-reliant high schoolers unable to reason, who's to say they'll still be able to empathize in the future?
Also, with the persistent lack of psychiatric services, I guarantee at some point in the future AI models will be used to (at least) triage medical mental health issues.
But LLMs - despite being extremely interesting technologies - aren't actual artificial intelligence like were imagining. They are large language models, which excel at mimicking human language.
It is kinda funny, really. In these fictions the AIs were usually portrayed as wanting to feel and paradoxically feeling inadequate for their missing feelings.
And yet the reality shows how tech moved the other direction: long before it can do true logic and indepth thinking, they have already got the ability to talk heartfelt, with anger etc.
Just like we thought AIs would take care of the tedious jobs for us, freeing humans to do more art... reality shows instead that it's the other way around: the language/visual models excel at making such art but can't really be trusted to consistently do tedious work correctly.
It seems these AI people are completely out of touch with reality.