"Verified by a human" - and that human will be an overworked, underpaid, overseas subcontractor who may well have an incentive to mash the "Confirm match" button from time to time to improve his performance.

Beyond that, are we seriously going to ignore the fact that eventually this system will share the private photos of someone's naked kid with some random subcontractor? How is that even remotely OK? Or that it will find and share actual CSAM with said subcontractor?

It only shares "visual derivatives" of images whose NeuralHash match the NeuralHash of known CSAM (either by being the same image ("perceptually") or a collision).

That to me just sounds like weasel words to avoid having to say that it shares images. Let's not beat about the bush, the "visual derivative" has to be good enough to identify what's going on in it for the manual confirmation.

Are you actually arguing in good faith here at all? Because I can't see how a "visual derivative" that's nevertheless good enough for manual confirmation is any better than the source image?

> Are you actually arguing in good faith here at all?

Are you? Because you just seemed to claim that it could match against innocent pictures of your naked children, but this tells me that you don’t understand that this system looks for known pictures, not for something that looks like naked children.

Edit: if you do, apologies, but then I’d say that Apple has suggested that it’s a low resolution version of the picture. This should be contrasted with server side scanning, where the server accesses all pictures fully.

This system does not use ML to find new CSAM images. It only checks for ones already in a known database. Your pictures of kids in the bathtub are not on the list.

What is show to the reviewer is a "visual derivative" which hasn't been clearly defined. A thumbnail image? Something with a censored section? We don't really know.

Yes I'm aware that it checks against a known database but clearly there can be collisions. So eventually it will share someone's private images.

It would need to have 30 collisions before anything even took place, which realistically isn't going to happen.

Most cameras nowadays can easily take a burst of 30 visually very similar images in a second.

Lots of people leave their cameras in burst mode.

Or rather, it will share some gray blob apparently.

What is the point of sharing a gray blob? How is that going to prove anything?

I think the gray blob refers to engineered hash collisions, like the example in the article link.

The "verified by a human" is only the the last step of the process for Apple. After that, it's given to NCMEC for review, then it's reported to authorities. The Apple subcontractor isn't the only one verifying things here. What you're saying is not realistic.

I find it very hard to believe that Apple will put literal future of iPhone (just imagine the amount of bad press if one false negative comes out of review process and is reported to the authorities) to "overworked, underpaid overseas subcontractor".

Apple is greedy, ruthless, tone-deaf machine, but I don't think they're stupid.

That’s already how app reviews work.

I think rejecting one of million apps and angering one developer is a bit different than reporting someone about a child pornography/abuse, ruining their life and making a global scandal that can put future iPhone sales in jeopardy.

Barely anyone ever cared about privacy. Convenience, no matter how insignificant, always wins. I will be surprised if this is the last straw that finally gets the masses to care about privacy. In reality there might be a few scandals but at some point no one will care anymore and Apple will remain popular.

> It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography.

I've never personally seen or looked for any images like this, but if they weren't already proliferating online and widely available to criminals, why would we need to build an elaborate client-side scanning system to detect and report people who have copies of them?

Do you have access to darknet child pornography trading networks? I don't.

OK, now I'm curious. How does darknet stuff actually work?

Is there some darknet search engine that you access over Tor and type whatever illegal thing you seek such as "child porn" or "stolen credit cards" or "heroin" or "money laundering" into and it points you to providers of those illegal goods and services reachable over some reasonably secure channel?

Or is it one of those things where someone already using a particular child porn or stolen card or money laundering or heroin selling site has to put you in contact with them?

Hmm. Tor onion addresses are generally super long and have a large random element to them, such as

facebookwkhpilnemxj7asaniu7vnjj.onion

where the "Facebook" part is there for convenience and may likely not be present for actual illegal addresses. As such, you probably won't be able to find a CP website unless you actively brute force the entire onion address space.

I don't know how the sharing actually happens, but I don't think a real search engine for the dark net exists. There are websites that try (Ahlia, I think?) but they are super limited in scope.

I suspect addresses for drugs and CP are shared person-to-person.

dark.fail indexes darknet markets, but all the ones I've seen ban CSAM

Agreed. This attack still has consequences for people but they will not run into any legal trouble as far as I'm aware.

I doubt someone who is able/wanting to attack such a system would be unwilling to own some of the material themselves.

That assumes that the human review process is competent, your own images aren’t poisoned in some way (consider your own kids in the bath with some noise added) etc.

In the mean time they lock your account which means your entire digital life stops dead until their review process is done.

No way do I accept any of this.

Also the hashes are on the device I understand and it’s not going to be that difficult to extract them.

You:

> the hashes are on the device I understand and it’s not going to be that difficult to extract them.

----

A Review of the Cryptography Behind the Apple PSI System, Benny Pinkas, Dept. of Computer Science, Bar-Ilan University:

> Do users learn the CSAM database? No user receives any CSAM photo, not even in encrypted form. Users receive a data structure of blinded fingerprints of photos in the CSAM database. Users cannot recover these fingerprints and therefore cannot use them to identify which photos are in the CSAM database.

https://www.apple.com/child-safety/pdf/Technical_Assessment_...

----

The Apple PSI Protocol, Mihir Bellare, Department of Computer Science and Engineering University of California, San Diego:

> users do not learn the contents of the CSAM database.

https://www.apple.com/child-safety/pdf/Technical_Assessment_...

----

A Concrete-Security Analysis of the Apple PSI Protocol, Mihir Bellare, Department of Computer Science and Engineering University of California, San Diego:

> the database of CSAM photos should not be made public or become known to the user. Apple has found a way to detect and report CSAM offenders while respecting these privacy constraints.

https://www.apple.com/child-safety/pdf/Alternative_Security_...

So fundamentally there’s a lot of mechanism around keeping secrets but the source material is available. Hmm.

https://youtu.be/eU2Or5rCN_Y

The human reviewer would be able to check against the exact image that generated the hash in the first place. Taking another completely unrelated image and perturbing it would be immediately obvious.

So there’s an office somewhere with computers full of illegal child porn that people are staring at and comparing your photos to?

There’s some irony in that.

Yes. That is called NCMEC in the US and it is a core aspect of how this whole process works.

If you don’t understand the details of this, I’ll recommend this podcast episode which sums it up and discusses the implications https://atp.fm/443

I am aware of the details. The episode of Brass Eye comes into context here, the relevant clip being as follows, showing exactly the issues of competence.

https://youtu.be/_U-7L1tmBAo

I don’t have much of an opinion here except that it is silly to write “I am aware of the details” when someone gives you a helpful explanation and you are obviously not aware of the details, as you had just asked a question betraying.

I’m sorry, what is your point? I legitimately don’t understand what you mean.

> In the mean time they lock your account which means your entire digital life stops dead until their review process is done.

Do you have a source for this, or are you just making things up?

the blacklisted hashes are openly given to thousands of companies by a foundation called the National Center for Missing & Exploited Children, in both PhotoDNA and MD5 versions of the hashes.

Yes, I can't give you an open link, but I imagine at least hundreds of thousands of people have access to download them. Some companies with better security than others

It only takes 1 guy to sell it to an NSO-type company, who would then use it to target attacks for $$$

Where would that $$$ come from? Like I said, the only attack possible is to overload the human reviewers, and even that is difficult to pull off except as a coordinated effort by lots of people.