Of course it's not natural in Europe and their are laws against this kind of thing.

The quality of the comments here are really poor.

In art class in an American high school it was common to discuss and share digital images of famous nudes. I’m sure this system won’t interfere with that at all because of course the only nudity people might see or share is bad, right? In case my sarcasm isn’t clear: of course it will interfere with art classes. Imagine if the art under discussion is photography? Then students and teachers will be sharing real nudes which are completely appropriate under the supervision of adults. Pornography like artistic creativity is contextual and the eye of the beholder. Here we see the eye is a black box of AI. This is horrifying. Likely it will mean this type of education simply won’t happen in these spaces. This is a direct and immediate limitation on free people.

The primary problem with censorship isn’t usually the core motivation. We can all agree that adults shouldn’t send porn to kids, and kids shouldn’t send nudes to anyone, etc etc. Sadly we don’t all agree on what that means - the law is a poor substitute for thinking and understanding context.

Automatic policing will lead to automatic prosecution, we have seen this with Facebook intercepting messages and automatically dispatching relevant authorities. In the case of suicide risks, many agreed this was a public good. Is it a public good when it is done for pot which is illegal federally even though it is legal in California? Do we really want a private company making these choices? Do we really want them in the middle even if it’s “on device” when the device isn’t even open for meaningful inspection?

It is the infrastructure and the corner cases which create a chilling effect on society. Once this is done in one place, it will expand to others.

Do you really think that Saudi won’t push for this on every woman’s telephone? Why limit it only to children? And why limit it only to being opt in?

Do you really not see the future you’re endorsing and that apple is building based on profit motives which are dressed up as “obvious” moral protection schemes?

Remember: the husband in Saudi is the legal guardian/censor of his children but also his fully grown (usually but not always) adult wife.

This can technically expand to textual rather than image censorship as well, we should stop it before that happens. Your device is your own personal space as much as the device is yours or as permitted by law (eg union organization on company time and company computers is sometimes protected).

Providing solutions to do automatic policing on your own device is insanely totalitarian. It isn’t a solution to doing it on company servers which also should not happen. We should not be building this technology and we should regulate the companies to uphold basic civil liberties, especially when they are a de facto monopoly.