News for hackers. Stuff that matters.
Re:A catalog of naturally occurring images whose Apple Neura... by saithound on Wednesday August 25, 2021 @06:54AM (#28300005)
Discussing the preimage attack on NeuralHash is not technical ignorance. Dismissing the preimage attack as irrelevant is.

0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever.

1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image.

2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No.

First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens.

Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs).

The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security.

Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly".

3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not.

[1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag...

[2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle

[3] https://news.ycombinator.com/item?id=28236102

[4] https://security.googleblog.com/2017/02/announcing-first-sha...

[5] https://www.zdnet.com/article/sha-1-collision-attacks-are-no...

Re:A catalog of naturally occurring images whose Apple Neura... by simondotau on Wednesday August 25, 2021 @07:03AM (#28300054)
That's a lot of words to say that you could sufficiently mangle an image that it could pass through all of Apple's algorithmic hurdles while not actually being CSAM. Of that I have no doubt. You could definitely generate a mangled image that fools multiple perceptual hash algorithms.

Let's set aside the questions of where you got all these hashes to generate collisions with, how you got 30 of these mangled images into your victim's camera roll without them noticing. And let's also set aside whether your victim's device is an iPhone with iCloud Photo Library enabled (and has sufficient storage). I still don't get what these mangled images have achieved, other than giving the manual review team something other than child porn to look at.

Seems to me like it'd be easier to just find actual child porn, print it out, place it somewhere in the victim's house and then report it to the police.

Re:A catalog of naturally occurring images whose Apple Neura... by dTal on Wednesday August 25, 2021 @07:12AM (#28300097)
I think you missed the point of the first paragraph. The point is that you can now hide child porn by making its hash collide with innocent images. They won't ever make it to manual review. Ergo, NeuralHash is now useless.
Re:A catalog of naturally occurring images whose Apple Neura... by simondotau on Wednesday August 25, 2021 @07:15AM (#28300122)
Why would anyone go to the effort of technical concealment[1] of CSAM when they could just resist the urge to import child porn in their phone's photo library in the first place? I've managed to resist the urge to import regular porn into my photo library, and being caught with regular porn is (at most) embarrassing. It's not potentially life-destroying.

It's inconceivable that anyone could desire possession of NCMEC-catalogued CSAM images without being aware that they're risking serious consequences if they're caught. Who wants their deepest, darkest, potentially life-ruining secrets just milling about with photos of the dog and last night's dinner?

[1] ...which is all but impossible for an average user to prove was effective; it's not like the Photos app has a "Not Child Porn!" checkmark.

Re:A catalog of naturally occurring images whose Apple Neura... by laszlokorte on Wednesday August 25, 2021 @09:22AM (#28301323)
I agree with you in that I do not understand why anybody doing something illegal would upload related data to a cloud storage.

But if nobody would import CSAM into their icloud library why do all the pictures need to be scanned in the first place? I would imagine anybody doing major illegal stuff being informed about important measures in order to not be caught.

Re:A catalog of naturally occurring images whose Apple Neura... by simondotau on Wednesday August 25, 2021 @10:01AM (#28301817)
I agree, why Apple is doing this is an interesting and pertinent question. I don't think it's actually because they think this will put a big dent into CSAM. So the question is: what is motivating Apple?

Perhaps it's a prerequisite for deploying end-to-end encryption of iCloud Photo Library and/or iCloud Backups. The latter in particular has remained decryptable by Apple supposedly due to pressure from the FBI. Perhaps CSAM is what the FBI are using to justify their pressure.

Perhaps it's because Apple's team of lobbyists are seeing ahead to future anti-privacy, anti-encryption legislation being justified under the guise of CSAM. If Apple can show that the CSAM problem is already "solved" then such justifications disintegrate.

Re:A catalog of naturally occurring images whose Apple Neura... by FabHK on Wednesday August 25, 2021 @08:32AM (#28300818)
So, the presumed attack (not against individuals, but to defeat the system) is

1. Identify some innocuous pictures that many many people have (memes, Beyoncé, whatever).

2. Produce CSAM.

3. Mangle it such that it is still CSAM visually, but NeuralHash-collides with the innocuous pictures from step 1.

4. Distribute.

5. Wait until they are (via some other mechanism) a) identified as CSAM, b) added to the NCMEC database, c) added to the Apple on-device database of blinded hashes in some iOS update.

6. Millions of people are suddenly incorrectly flagged for exceeding the threshold by NeuralHash (since they have the innocuous pictures in their library), and the review teams are flooded and can't pick out the small number of actual CSAM holders.

That is not without a certain elegance. However, it seems to me that

A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.

B) it would be quickly defeated by amending the 2nd tier algorithm (between NeuralHash and human review), though, as you highlight, that might be tricky given that the team working on this presumably only has access to the innocuous false positive collision image, not the (purposefully mangled) CSAM.

Re:A catalog of naturally occurring images whose Apple Neura... by saithound on Wednesday August 25, 2021 @08:46AM (#28300926)
> A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.

Note that this requires no single "desired" target picture. There are millions of popular, innocuous pictures. As long as you can make your CSAM match any one of them without significant mangling, you're good to go. Not having to choose one specific target makes this much easier to accomplish.

Re:A catalog of naturally occurring images whose Apple Neura... by nullc on Wednesday August 25, 2021 @02:27PM (#28305107)
> it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.

You can. Here is an example I created (with links to more): https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...

I'm so tired of people suggesting that you can't. Please explain to me why you posted suggesting otherwise.

I've contemplated making some that are also photodna matches, I expect that it's possible. But access to photodna is only through some awful windows tools, and AFAICT people would just keep posting denials even after an example was posted-- so it's not worth the effort at least not worth it just to further the public discussion.

Re:A catalog of naturally occurring images whose Apple Neura... by FabHK on Wednesday August 25, 2021 @05:27PM (#28307268)
> Please explain to me why you posted suggesting otherwise.

a) I didn't suggest otherwise, I said that it is predicated on that assumption, about which I was undecided, largely because b) I didn't know better.

I read that thread 7 days ago, when the collisions were a gray blob or a clearly modified dog (to Lena) or clearly modified Lena (to dog). I hadn't re-read the thread in the last 4 to 5 days, when you demonstrated the natural looking collisions (second-preimage images).

Very impressive work I wasn't aware of.

Re:A catalog of naturally occurring images whose Apple Neura... by nullc on Wednesday August 25, 2021 @05:34PM (#28307353)
Great answer. Thank you!
Re:A catalog of naturally occurring images whose Apple Neura... by SiempreViernes on Wednesday August 25, 2021 @09:05AM (#28301135)
There's also the _very_ important assumption that Apple doesn't check for collisions with popular images before adding new things to the database.
Re:A catalog of naturally occurring images whose Apple Neura... by yeldarb on Wednesday August 25, 2021 @07:31AM (#28300252)
> allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images

That’s a really interesting attack vector I hadn’t seen mentioned previously.

Most people are talking about the potential for adversarial images to be sent to users. If they were instead injected into the database itself (either by poisoning real CSAM or social engineering) that would have far wider ramifications.

I wonder what the most widely-saved pornographic images are across iCloud users.

If actual CSAM were perturbed to match the hash of, say, images from the celebrity nude leak a few years back and added to the database then thousands of users could be sent to “human review”. Since the images are actually explicit how would the human reviewers know not to flag them to authorities?

Re:A catalog of naturally occurring images whose Apple Neura... by theshrike79 on Wednesday August 25, 2021 @08:37AM (#28300847)
Because they're not CSAM?

People don't seem to grasp what kind of images end up in the CSAM databases. They are most definitely not "leaked celebrity nude selfie" level stuff.

Think of the most vile sexual thing you could do to a child and then times that by two and halve the child's age in your mind. That's the shit that gets in there.

It's not something even 4chan weebaboos share. It's stuff that makes Liveleak regulars go "ewwwww, gross".

Re:A catalog of naturally occurring images whose Apple Neura... by saithound on Wednesday August 25, 2021 @08:49AM (#28300965)
The issue is not that a celebrity photo might get into a CSAM database. It's that a true CSAM photo, which has been modified to have the same hash as a "leaked celebrity nude selfie", will probably get into a CSAM database.
Re:A catalog of naturally occurring images whose Apple Neura... by theshrike79 on Wednesday August 25, 2021 @12:31PM (#28303664)
The CSAM database is not generated by an AI. Actual people hand-pick the worst images to include in the list.
Re:A catalog of naturally occurring images whose Apple Neura... by yeldarb on Wednesday August 25, 2021 @09:58AM (#28301762)
How is the human reviewer expected to make that judgement call? They’re not allowed to see the original to compare with for obvious reasons.
Re:A catalog of naturally occurring images whose Apple Neura... by theshrike79 on Wednesday August 25, 2021 @12:29PM (#28303641)
They see a "visual derivative" of the original.

I think they can tell the difference between a naked celebrity and a molested child.