Apparently not that hard: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue....

See also here: https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29...

While super impressive I haven't seen the thing that would actually destroy the algorithm which is given a hash and random reference image produce a new image which has that hash and looks like the reference image.

What you've seen is worse than that.

All you need to do to cause trouble right now, would be to get a bad image, hash it yourself, make a collision and distribute that.

Let's say for the time being that the list hashes themselves will be server-side. You won't ever get that list, but you don't need it in order to cause a collision. You would need your own supply of CSAM to hash yourself, which while distasteful is clearly also not impossible.

It might be useful to read the threat model document. Associated data from client neural hash matches are compared with the known CSAM database again on the server using a private perceptual hash before being forwarded to human reviewers, so all such an attack would do is expose non-private image derivatives to Apple. It would likely not put an account at risk for referral to NCMEC. In this sense, privacy is indeed preserved versus other server scanning solutions where an adversarial perceptual hash (PhotoDNA is effectively public as well, per an article shared on HN) would trigger human review of all the account’s data.

I assume these "non-private image derivatives" are downscaled versions of the original image. But for downscaling there also are adversarial techniques: perturb an image such that the downscaled version looks like a target image. See https://bdtechtalks.com/2020/08/03/machine-learning-adversar...

For the server side component, the challenge is knowing the target of adversarial generation since, again, the perceptual hash used is secret. At that point, you are reduced to the threat present in every other existing CSAM detection system.

> the perceptual hash used is secret

Yes, although I'm sure a sufficiently motivated attacker can obtain some CSAM that they are reasonably sure is present in the database, and generate the NeuralHash themselves.

> At that point, you are reduced to the threat present in every other existing CSAM detection system.

A difference could be that server-side CSAM detection will verify the entire image, and not just the image derivative, before notifying the authorities.

> Yes, although I'm sure a sufficiently motivated attacker can obtain some CSAM that they are reasonably sure is present in the database, and generate the NeuralHash themselves

Remind us what the attack is here? The neural hash and the visual derivative both have to match for an image to trigger detection.

I believe something like: an adversary creates an innocuous photo where:

* The photo itself is benign.

* The photo’s NeuralHash matches known CSAM.

* The photo’s image derivative is not benign. It looks visually like CSAM.

* The photo’s image derivative matches known CSAM per a private perceptual hash.

The above, combined, could have a victim reported to NCMEC without being aware they were targeted. Since Apple operates on image derivatives, they could be fooled unlike other cloud providers. That is the claim.

At that point, the victim could point law enforcement to the original CloudKit asset (safety vouchers include a reference to the associate asset) and clear their name. However, involving law enforcement can always be traumatic.

Sure, but did I miss something that suggests that all 4 of those conditions can actually be met? That seems like the part that has been made up.

Afaik the image derivative isn’t checked for looking like CSAM. It’s checked for looking like the specific CSAM from the database.

CSAM in the database is confidential, so the human reviewer just needs to be plausibly confident that they have CSAM in front of them. However, it’s not clear to me that you can pull off all three simultaneously. Furthermore, the attack doesn’t specify how they would adversarially generate an image derivative for a secret perceptual hash that they can’t run gradient descent on.

If someone wanted to plant CSAM and had control of an iCloud account, it seems far easier to send some emails with those images since iCloud Mail is actively scanned and nobody checks their iCloud Mail account, especially not the sent folder.

CSAM in the database is confidential.

The question is whether the visual derivatives are checked against derivatives from the database or just against abstract criteria. That seems to be an unknown.

> However, it’s not clear to me that you can pull off all three simultaneously

Agreed. People here seem to keep assuming that you can, but so far nobody has demonstrated that it is possible.

> before being forwarded to human reviewers

Does that mean that Apple employs people who manually review images known to be child pornography 9-to-5? Is it legal?

Yes, and so does every other major cloud service provider. The working conditions of these people are notoriously difficult and should be the subject of attention.

I'd imagine the key is that it's "manually review images suspected to be child pornography". The point of the review process is presumably that there are possible hash-collisions / false-positives, so the reviewers are what cause that transition from suspected to known.

Privacy is preserved by a team of humans looking at your private photos. Sounds like a good deal. What are we preserving again?

See https://news.ycombinator.com/item?id=28105849, which shows a POC to generate adversarial collisions for any neural network based perceptual hash scheme. The reason it works is because "(the network) is continuous(ly differentiable) and vulnerable to (gradient-)optimisation based attack".

Spy agencies could add CSAM images adversarially modified to match legit content they want to find. Then they need to have someone in Apple's team to intercept the reports. This way they can scan for any image.

That reminded me of the AT&T Room 641A

https://en.wikipedia.org/wiki/Room_641A

If this is the level that a "spy agency" is going to get involved, they would already skip all this BS and just upload the images directly to themselves.

Or for criminals to generate perceptually similar illegal images that are no longer triggered as a 'bad' hash.

So, these adversarial collisions are the images that I need to send to my enemies so that they go to prison when they upload those images to iCloud? It seems trivially easy to exploit.

You can technically hide an adversarial collision inside a complete legit normal image. It won’t be seen by human eyes but it will trigger a detection. In addition, you can do the complete opposite by perturbing a CSAM to output complete different hash to circumvent the detection. All of these vulnerabilities are well known for perceptual hash.

So right there seems to be an issue to me. It seems like if you were trading in CSAM, you would run CLEANER -all on anything and everything. Because you know someone has already written that as proof of concept here.

You can also just send your enemies CSAM, it is more effective at imprisoning them.

They probably would not hold onto them and also now you have a paper trail of sending CSAM to people. But if you were to alter some innocent looking photos and send them to someone, they might store those.

Yes, but that won’t work since innocent looking photos won’t match the visual derivative.

That sounds like a good reason not to create a virtual SWATing vector then.

No. Because, as documented in the published material, hits are also reviewed by humans.

Downscaled derivative images are what are checked, and if the reviewer even suspects that the downscaled image contains sexual content of a minor, they're required to report it. At that point, it is the authorities who decide whether or not to continue to investigate and determine if the user possesses CSAM. That investigation alone can ruin lives.

The "downscaled" image 360x360. It's pretty easy to compare two images at that size to determine if they are the same. You're not going to end up investigated unless you actually have actual picture of child pornography. Also, not just one, the process doesn't even start until you have a fairly large number.

It's highly unlikely that the apple reviewer will have access to the actual image in the database, instead they will assess whether the blurry,b/w,etc modified thumbnail could possibly be illegal. Given the level of secrecy surrounding the contents of the DB, even the normal NCMEC reviewers may not have access to those for comparison. There are plenty of CG images, small legal models, or even images of erotic toys which could be used as the base image which would look very suspicious as a low quality thumbnail.

There will be someone along the line who will check that the image is, in fact, the actual listed image. Otherwise, there is no legal case to be made.

> There will be someone along the line who will check that the image is, in fact, the actual listed image.

That someone will be law enforcement, and they will get a warrant for all of your electronic devices in order to determine if you actually have CSAM or not. It's literally their job to investigate whether crimes were committed or not. Those investigations alone can ruin lives, even more so if arrests are made based on the tips or suspicions.

I wonder what scenarios are facilitated by sending adversarial collisions over simply sending over CSAM.

The images are sent for manual verification before you're turned in, so no.