Yes, they can analyze it in the app on user devices before it is encrypted and transmitted. The app on user decide need access to clear text in order to encrypt the message. Same on the receiving end. The app on user device can analyze the message once it is decrypted on user device.

Wow did I not read something similar last couple of days about on-device machine learning being better than in the cloud machine learning and how Apple got that right. That's where this may also be going then if we follow your train of thought.

On device is where you want it if it's going to analyse really private data, or something effectively your own (such as homomorphic encryption, or a link to your own computers elsewhere).

You're likely to feel so much happier, freer and easier sharing your most personal life datastream with an AI assistant, if you can be sure its most intimate analysis is just between the two of you.

It puts me in mind of the two kinds of AI in Manna: https://en.wikipedia.org/wiki/Manna_(novel)

Coincidence or not, the dystopian AIs are somewhere in the cloud and work for someone else, while the utopian AIs are intimately personal to each user and work just for the user.

Sure why not? As long as no data ever leaves the device unencrypted and the encrypted data can only be decrypted by the client at the other end. Of course you'd probably have to take the app's word for it that that's actually what it's doing if you don't have the source, but that's no different from current E2E encryption offerings from WhatsApp etc.

The part I'm not sure about is whether the on-device certification that the message is "clean" couldn't be (easily) spoofed. But it would probably help curb distribution of illegal material anyway.

No, obviously not. The mental gymnastics involved here are impressive: the point of E2E encryption is to stop the service provider seeing or tampering with your messages. If they do that anyway it doesn't really matter how it's implemented. They could also just use a broken random number generator, or many other ways to implement the policies whilst still having encryption code in the product. It's the end result that matters, not the precise means of implementing it.

Phew, agreed. I mean of course the company "can" read the message. If it does, I would love to see that shown by the app upfront, so I can avoid using it.

Analysis happens on either end, not the network or servers. Of course if both ends are "cracked" this doesn't work, but the goal is to stop mass spread of disinformation. Most people won't modify their client.

Of course the party that is doing the encryption can see the original content...