> If you want to store your photos on Apple servers, shouldn’t they have the right to exclude CSAM content?

This seems worded to get a Yes answer. So, yes.

It's a big deal because it's unprecedented (to my knowledge) outside of the domain of malware*. Other cloud providers run checks of their own property, on their own property. This runs a check of your property, on your property. That's why people care now. The fact that this occurs because of an intention to upload to their server doesn't really change the problem, not unless you're only looking at this like an architectural diagram. Which I fear many people are.

A techie might look at this and see a simple architectural choice. Client-side code instead of server-side. Ok, neat. A more sophisticated techie might see a master plan to pave the way for E2EE. A net-win for privacy. Cool. But the problem doesn't go away. My phone, in my pocket, is now checking itself for evidence of a heinous crime.

*I hope the comparison isn't too extra. I was thinking, the idea of code running on my device, that I don't want to run, that can gather criminal evidence against me, and report it over the internet... yeah I can't get around it, that really reminds me of malware. Not from society's perspective. From society's perspective maybe it's verygoodware. But from the traditional user's perspective, code that runs on your device, that hurts you, is at least vigilante malware, even if you are terrible.

> From society's perspective maybe it's verygoodware.

At least in the US, the historical distinction between verygood searches and mal searches is given in the 4th amendment: "no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."

Of course that has been twisted and stretched before, and this is more of the same. But if literally everything is being scanned, "probable cause" is completely absent from the process. It is a fishing expedition of the exact type that the 4A was designed to prevent.

> "Other cloud providers run checks of their own property, on their own property. This runs a check of your property, on your property. That's why people care now."

iTunes Match is an iCloud service which (if you buy it and opt-in) scans your local on-device music library for copyrighted songs, tells Apple you have them, and then they let you listen to high quality versions of those songs on all your devices. And it's not filename or id3 tag matching, it's doing a fuzzy match that can identify the same song in low quality rips and in different file formats. It would be concievable for them to scan for banned audio lectures, or scan your whole device outside the iTunes library, or change it to check for other copyrighted files e.g. movies. It could concievably be reporting you to the MPAA/RIAA if it finds certain songs along with your public IP address so they can check if that IP address has ever been logged as torrenting those songs. It could "in future" be changed to look for and report evidence of torrenting or movie copying. There's nothing technical or regulatory(?) stopping Apple from saying "people with CSAM on their computers can't use iTunes Match" and making it scan the computer as a condition of use, is there? There's nothing technical stopping a government from asking "can your iTunes Match scan engine report video files in the iTunes library which match popular Tiannamen Square video MD5 hashes?", is there?

I do get that these are not the same seriousness, iTunes Match isn't (so far as we know) scanning to report crime but in so far as "Unprecedented on-device scanning for known content using an opaque database and a closed-source fuzzy-matching engine, it would only take a small change to make it look for other things, governments will definitely pressure them to do that and since they willingly built this system they will definitely agree, and all you can do is trust them", are they not samey enough to be relevant?

I don't understand your argument. You say people care about this because it "runs a check of your property, on your property". Would iCloud servers running this check after the photos are uploaded to iCloud be any better?

> My phone, in my pocket, is now checking itself for evidence of a heinous crime.

I see your point here - this is a slippery slope for Apple. However I don’t see how anyone could achieve both purposes - no fingerprint reporting and prevention of CSAM storage on Apple servers.

Also, a practical thing to do is to just not store your photos on iCloud but use something else for sync and backup - there might be a startup opportunity here if enough people care.

Personally I don't see on device scanning as significantly different than cloud scanning. I think the widespread acceptance of scanning personal data stored on the cloud is a serious mistake. Cloud storage services are acting as agents of the user and so should not be doing any scanning or interpreting of data not explicitly for providing the service to the end user. Scanning/interpreting should only happen when data is shared or disseminated, as that is a non-personal action.

If I own my data, someone processing this data on my behalf has no right or obligation to scan it for illegal content. The fact that this data sometimes sits on hard drives owned by another party just isn't a relevant factor. Presumably I still own my car when it sits in the garage at the shop. They have no right or obligation to rummage around looking for evidence of a crime. I don't see abstract data as any different.

What does the law say, though? Possession of CSAM by any organization or person other than NCMEC is flatly illegal. Even other branches of government, including law enforcement, may not have any in their possession. My question is -- does CSAM residing on Apple's servers, even when it is 'owned' by a customer, count as them possessing it? What about if it is encrypted?

Should every minecraft server be checking if any arbitrary sequence of blocks on that server can encode to a binary representation of CSAM which when hashed matches something in the NCMEC database?

You could argue that a minecraft server is technically in possession of CSAM if that's the case, but you could spend an infinite amount of money looking at various possible sequences and are bound to find many more false positives than true positives.

Services should have a duty to report CSAM when they notice it, but the lengths they should go to search for CSAM should be limited by cost/benefit and privacy concerns.

My impression is that letting people upload CSAM to a cloud service has no positive benefit because of the supposed link between CSAM consumption and CSA, and it carries a very high risk of criminal liability, so there's no incentive for companies to completely ignore the files that users upload. Otherwise, people will eventually notice and the service will be denounced as "pedophile-friendly," and then the law will take notice and force them to give up the data.

This type of scenario is what happened with the messaging service Kik, which was reportedly used to distribute CSAM in private chats. Law enforcement agencies said the company wasn't providing timely responses and that children were being actively abused as a result. This is about as damaging of an accusation you can leverage against a company.

Laws against CSAM worldwide are not going away for good reasons, so there is always going to be a justifiable argument that storing certain classes of data is illegal. Hence, anyone wanting to run a cloud service that stores user data will have to obey by those laws, regardless of how proactive they are in scanning for the material. Absolute privacy in the cloud is impossible to achieve with those rules in place.

(Edit: it seems like the algorithm does not work like I thought it did, you can basically disregard this comment.

This has been mentioned on here before, but it's known CSAM possession that's illegal. Apple keeps your files encrypted until its algorithm thinks your encrypted file is too similar to CSAM, and then it decrypts it and sends it to Apple for review. There's a few things here.

- The algorithm is a black box, so nobody knows how many false positives it hits.

- Apple's willingness to decrypt files without the consent of the owner makes the encryption seem like a bit of a sham.

- I imagine many are skeptical of Apple's ability to judge CSAM accurately. If I take a photo of my kids in a bathtub, is that CSAM? What about teenagers in a relationship sharing nudes. The law is a blunt and cruel instrument, and we've gotten away without hurting too many innocent people so far because the process is run by humans, but computers are not known for being gracious.

> The algorithm is a black box

So we know for sure they're not just using PhotoDNA?

> If I take a photo of my kids in a bathtub, .....

Kinda the same question. If they're using PhotoDNA, then that's not really a risk, right? Isn't this technology well understood at this point?

Looks like you're right. I edited my comment. It looks like there's a couple fairly different changes that are happening:

- There's a system to catch CSAM that is either PhotoDNA or something that works similarly.

- There's a system to detect novel nudes, and notify parents if their children view them.

I think I got these two mixed together.

> I think I got these two mixed together.

That's fair. Apple did a shit job of explaining themselves, and it has been compounded by a lot of misinformation (deliberate or not) in response. I'm trying really, really hard to moderate my reaction to this whole mess until I feel like I actually understand what Apple intends to do. I don't make platform jumps lightly.

If it is encrypted, the data storage provider has no chance to know what it is.

Which is exactly why these policies are so dim witted.

Dragnet violation of everyone’s privacy while anyone even remotely sophisticated can easily evade it by just encrypting the data upfront.

Ben's Stratechery article explains the distinction:

> (f)Protection of Privacy.—Nothing in this section shall be construed to require a provider to—

(1) monitor any user, subscriber, or customer of that provider; (2) monitor the content of any communication of any person described in paragraph (1); or (3) affirmatively search, screen, or scan for facts or circumstances described in sections (a) and (b).

https://stratechery.com/2021/apples-mistake/

According to Krawetz's article, Apple is not allowed to scan the files on their servers without a warrant. That's why they want to run the scan on your phone instead.

But at least you sent them away and can have reasonable expectation to be betrayed. But that your camera device also scans and report what it doesn't like.

Because if the content is entirely encrypted(like apple says it is) they aren't legally liable and it's entirely voluntary that they do this.

Also, no one(well, most people) has any issue with photos being scanned in the iCloud. Photos in Google Photos have been scanned for years and no one cares. The problem is that apple said that photos are encrypted on your device and in the cloud, but now your phone will scan the pictures and if they fail some magical test that you can't inspect, your pictures will be sent unencrypted for verification without telling you. So you think you're sending pictures to secure storage, but nope, actually their algorithm decided that the picture is dodgy in some way so in fact it's sent for viewing by some unknown person. But hey don't worry, you can trust apple, they will definitely only verify it and do nothing else. Because a big American corporation is totally trustworthy.

I mean even the verification is problematic. At no point I want an "certified" Apple employee or another parent looking at naked pictures of my kids, for example.

But there's nothing in this proposed implementation that could ever possibly result in that, because random pics of your kids would not be in a database of known CSAM content. So your pics wouldn't match the hash values.

I'm constantly surprised how even people on HN are confused about this - read the white paper apple published. It very explicitly says that they are using a perceptual(similarity based) hash, and we(well, not me specifically, researchers) have demonstrated that it's trivial to produce a picture that isn't even remotely similar in theme but still produces the same perceptual hash.

Apple's solution to this problem is that their employee will actually verify the picture before sending it to authorities. Which again, is one of the problems people have with this system.

If that hash stuff actually works (identifies CSAM with high probability), then that puts Apple in the position of knowingly possessing CSAM. The "One Bad Apple" article a couple days ago mentions that their staff was sometimes exposed to CSAM but that it was unknowing, because the prior probability of an image they looked at being CSAM was quite low. If the probability was high (like > 50%?), it wounded like they would have been obligated to send it to NCEMC without looking at it.

https://www.hackerfactor.com/blog/index.php?/archives/929-On...

So you are saying the hash values can never result in a collision? That in fact there is literally zero chance that two different images could result in the same hash?

Because that doesn’t sound correct to me…

It is essentially true that a cryptographic hash can never collide. But what Apple claims to be using is a perceptual hash. Perceptual hashes can not just be scaled wider trivially, to reduce probability of collision, like cryptographic hashes can.

So in other words, there is a chance for collision. And when (not if) there's a false positive, eventually the system will transmit private information to some underpaid army of people who will be receiving either real child porn or pictures of someone's naked children for confirmation.

Ignoring the nasty aspects of doing that kind of work, I don't see any way to buttress the fact that Apple has taken another step on the universally one way street of ever increasing surveillance. And whataboutism in the form of "oh but other cloud providers are already doing this" is an extremely weak argument because I don't want to use other cloud providers. I liked Apple, because irrespective of their real motives (ie money), they seemed to be privacy focused. The backhanded way they tried to sell this "feature" shows that they are just as bad as the others.

The issue is that the scanning happens on your device just before upload. So now your own device is scanning for illegal activity _on_ your phone not the servers.

The second issue is that it will alert authorities.

In regards to CSAM content those issues may not sound terrible. But the second it is expanded to texts, things you say, websites you visit or apps you use it's a lot scarier. And what if instead of CSAM content it is extended to alert authorities for _any_ activity deemed undesirable by your government

Logically the next step is to scan for any copyrighted content and notify authorities that you're watching a movie without paying for it. After all, it's all about catching criminals, how could you possibly object.

That's a fairly large step, though. Apple cares first and foremost about their reputation. If this feature catches a real predator, it is 100% good PR. Every single false positive that makes it into the news is a huge loss, which strongly incentivizes them to avoid that. The last thing I expect them to do is expand the risk surface for something as trivial as copyright enforcement.

> Every single false positive that makes it into the news is a huge loss, which strongly incentivizes them to avoid that.

Just to be clear, "false positive" in this case means an innocent person is accused of trafficking in child sexual abuse material. It's likely they will be raided.

Sure, that's bad if you're Apple, but it's a lot worse if you're the alleged predator.

The thing (many) people are worried about is that, having instituted a backdoor on the user's device, Apple may lose the ability to control how it is used. In the past, their strongest defense when the FBI asked them to use a backdoor to decrypt a user's device password has been "we don't have a backdoor that allows us to decrypt a user's device password, so we couldn't do that for you even if we wanted to".

From now on, when asked to check whether a user's encrypted phone contains arbitrary content the FBI wants to know about, Apple can no longer say "we don't have a way to do that." Sooner or later, you can bet they will start doing it, whether they want to or not.

CASM is already a pretty huge step into legal gray areas. There is no guarantee by anyone that images in the database are illegal. If the database could be guaranteed to only include real photos directly showing sex with a minor I personally would be all for device side scanning even without upload. However who knows what is considered CASM. I would not be at all surprised if it included Guess ads and cartoon parody. Heck in Australia it is illegal to share pictures of adult topless women with small busts because people may pretend it is child porn.

If this feature leads to anyone losing their job due to incorrect criminal accusations it will not even make the papers because we expect the accused are guilty anyway. Apple won't shed a tear until there is a class action.

Except they tout needing multiple matches before they do anything about it.

Which could also be spun as "Apple allows X freebies of known/highly suspected CSAM on your device before they'll tell anybody".

The amount of PR failure that has gone into all this is huge and multi-level.

You're intentionally running a device with a Digital Rights Management module, so...

I’d expect a secure and privacy focused cloud data storage provider to not know what I’m storing _at all_.

Let’s not beat about the bush, if someone wants to store information in a form that can’t be decrypted by Apple, they can. This is a stupid dragnet policy that won’t catch anyone sophisticated.

Apple focused the last years pitching themselves as the tech giant who actually cares about privacy. They seemed to be consciously building this image.

To now implement scanning of private information and then try and sell this obvious 180degree slippery slope turnaround in the most weasel worded “but think of the children” trope is an insult to the customers’ intelligence.

I was a keen Apple consumer because I felt that even if their motivation was profit, this was a company who focused on privacy. It was a distinct selling point.

I certainly won’t be buying more Apple products.

For me, Apple lost the main reason to buy their stuff. If they are going to do the same thing everyone else is doing, I refuse to pay the premium they charge.

If they were scanning images that were uploaded to icloud on Apple's servers, no one would care. iCloud is not encrypted and Apple provides governments access to iClod data, everyone knows that, and other cloud providers already scan content for CSAM material. The difference is that Apple is doing this scanning on your phone/computer. Right now, they say that only images that uploaded to iCloud will be scanned, but what's to stop them from scanning other files too? There's been a lot of pushback because this is essentially a back door into the device that governments can abuse.

Apple can do anything they want on every iphone, always have, and always will. Whether this feature exists or not changes that in no way, their technical ability to snoop through everyone’s stuff is the same. So far they’ve shown restraint with that ability.

I think what people are getting riled up about is not the technical ability, it’s the lack of restraint, the willingness to search through everyone’s personal stuff on their phones. This is like the cops sending a drug-sniffing dog into everyone’s home once a day, with the excuse that it is privacy-preserving because no human enters the premises, and that only truly bad people will get caught. There is a difference between scanning in the cloud and scanning on device. One is looking through your stuff after you’ve stored it in a storage unit, and the other is looking through your stuff while it is still in your home. Apple’s excuse is that you were going to move it anyway, but somehow that doesn’t actually excuse things.

How can Apple scan encrypted photos?

I should have said, "iCloud is not end-to-end encrypted". Apple has full access to everything you upload to iCloud, because they control the encryption keys, not you.

Apple has the keys to decrypt them.

Note how they use your device to do the dirty work for them, instead of doing what everyone else is doing and scanning stuff on their servers.

Because this way they can encrypt things on your phone and claim that they can't see your photos once on the server (because whatever they had to do with those photos, was already done on your own phone).

(There are many ways this can be a slippery slope, but we don't have to pretend they could just so what ever body else is doing just as easily and they just want to do it on your phone because they are lazy or whatever. This is a solution to a legitimate problem and also it turns out that people are rightfully worried about what's next; those two facts can coexist)

Because they don't want to see your private files.

While apple owns the servers they shouldn't be legally lake. No more than a self storage facility is liable for the items individuals sure in their units.

I'm not so sure that is true anymore. FOSTA-SESTA makes them liable, I think?

FOSTA-SESTA is all about "knowingly" participating in "sex trafficking". a) I'm not sure that you can knowingly have an idea what's in encrypted blobs, so I don't think this would enable E2EE; they seem orthogonal. b) They're not policing a communication medium, but local files. Sex trafficking almost tautologically implies some kind of communication, so the target being local files instead of, say, local storage of apps like messenger and signal seems a bit off as an explanation.