You're likely to feel so much happier, freer and easier sharing your most personal life datastream with an AI assistant, if you can be sure its most intimate analysis is just between the two of you.
It puts me in mind of the two kinds of AI in Manna: https://en.wikipedia.org/wiki/Manna_(novel)
Coincidence or not, the dystopian AIs are somewhere in the cloud and work for someone else, while the utopian AIs are intimately personal to each user and work just for the user.
The part I'm not sure about is whether the on-device certification that the message is "clean" couldn't be (easily) spoofed. But it would probably help curb distribution of illegal material anyway.
This is stupid. Lots of naked baby photos get sent in my culture (Eastern-European country) in a most non-harmful way, i.e. from parents to the kids' grand-parents or even to the parents' close friends (especially from the mother to her friends). Your supposed filter will most probably block that social-sharing process (because it will see photos of naked children => very, very bad), not realising the above mentioned context.
Truth is truth. Saying something is possible isn't the same as advocating that it be done, and it's useful to point out something is possible when people at first seem to think it is not.
Also, I was talking about messages, not baby photos, and with regard to misleading or false information, hoaxes etc that cause people to behave more dangerously to others during a pandemic. Saving lives, that sort of thing.
If it's giving the user advice that others have judged what they are retweeting to be a hoax or bad medical advice, that's not blocking, it's providing context. If they don't like it, they should be able to dial it down.
With regard to baby photos, if a network starts blocking those due to poor filtering, I would hope people switch over to another network that lets them share the photos.
> Your supposed filter will most probably block
I wasn't talking about filtering particularly, the emphasis was on providing a note to the user. Much like when Twitter attached a note to Trump's tweets.
In any case, the analysis I had in mind is not "skin tone filters" and that sort of nonsense. It's not meant to be thought police, working for someone else. It's meant to advise the users themselves to think again about some content. At least at the currently level of sophistication, that would be "we recognise this particular message or photo".
There are better and worse ways to implement it of course.
(Although, something that blocks communication (which I don't think I agree with anyway) based on local analysis is an autonomy violation. But not a privacy violation.)
Something that tells you when you've just received a well documented hoax is not malware, it's probably useful, and most people will probably keep it switched on if the quality is consistently good.
By your logic, spam filtering (outgoing and incoming) is also malware, and a privacy violation. (Even though it protects people against malware, and indirectly protects privacy.)
Do you believe spam filtering is bad? I doubt it.
Yes, people ask for certain kinds of anlysis based message blocking all the time. We begged and pleaded for better spam filtering 20 years ago because the vast majority of messages were pure spam and it had made email difficult to use. It's a major reason people switched to Gmail, because other providers' spam filters weren't good enough.
I think the key feature most people would want in any kind of alerting, tagging or filtering is that it does what they want, rather than what the enemy wants, as it were. As people's preferences differ, that can only happen if it's configurable by them rather than blanket imposed. Things like ad blockers work this way - you can change the defaults if you want - and people seem to like those.
Several commenters point out end-to-end encryption would prevent filtering or tagging messages.
But that's not true. The message analysis could be done at either endpoint without violating privacy.
Tagging (or removing) a message before you send/forward it, or after you receive it, with "the central message of this comment has been tagged as "probably a hoax" by hoaxtracker.com; check out this CDC notice <here> to learn more".
<here> does not need to be a URL which reveals much other than your general interest in the subject. But if that seems too revealing, it could already be already available as part of the endpoint's filtering data and readable locally.
Lots of people forward (retweet), or write a little something before resharing what is false or misleading information, not realising they're doing so. I would not be surprised if getting those tags, rarely enough to stand out, before they send the message would cause some people to hesitate and check/think a bit more before sending. Maybe rephrase their attached comment into a question rather than confident outrage.
Technically this is not much different from privacy-preserving spam filtering.