If it's a critical part of the system, then it should be inspected thoroughly. If Apple claims a minuscule chance of a hash collision, and the reality is that collisions are relatively common, that significantly changes the requirements for the backend system, which Apple keeps secret. We have every right to believe, bbased oon ppublic info, that Apple was expecting that NeuralHash would be almost fool-proof, leaving the backend system to be a rubber stamp. This would be tragic.
Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the original post here. In fact, I've not seen anybody look at the hash distribution over natural images, which would be an actual argument against the system.
Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about)
This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)
Apparently they have not - They say facebook reported something like 20m images in a year, and Apple reported 250.
Eroded trust, sure, but that is mostly because they did a terrible job communicating it.
The whole point for this is to be a probabilistic filter, so that they need to run the real CSAM scanner on a subset of files.
You can fall into two camps:
a) apple should never ever scan my private images I upload on their cloud. b) apple can scan the images once they reach their servers.
If you pick (a), then clearly neuralhash shouldn't exist and you can argue against that on the ground that you want utter privacy. But you have to be consistent:every other cloud service that does scan the images server side should receive the same critique.
If you pick (b), then you must recognize that this additional machinery doesn't increase their reach to your private data, but quite the opposite, it allows them to implement e2e encryption for 99.9% of your content. You may argue that it's unnecessary and confusing and spooky and be afraid of the slippery slope precendent for other uses.
FB and Google will exhaustively analyse every single facet of your online presence and use your pictures to train their ML models for face detection and object detection.
Apple, on the other hand, even explicitly splits Map directions to segments so that they can't know where you left from and where you are going to.
Anyway how is your assumption make sense , Apple cares about children and about your privacy so scanning your images in iCloud was wrong until 2021 when something changed, what changed? does Apple cares more about children starting from now or they care less about privacy? or are they forced to do it?
Therefore, why trust any of their other claims?
Just because someone has found an image of a nearly featureless diagonal thing which collides with another image of a nearly featureless diagonal thing, that doesn't disprove Apple's claims.
Given people can now generate images that collide, it seems like the statistical likelihood has drastically changed since it was originally announced.
So again the actual argument becomes: what is that distribution like?
I doubt that's what they did. I think they ran tests on huge numbers of pictures, got an estimate, put in a safety factor, and determined the threshold to hit their target (and put in another safety buffer then).
Naturally occurring collisions are not going to be an issue, and adversarial ones neither, I predict. Just as with current cloud providers.
Apple never made a false claim. They have never anywhere stated that neuralhash makes false positives at 1 in a trillion. Only that that is the rate for the system as a whole to flag accounts for review. The explicitly mention that they will vary the number of matches needed to maintain this if it turns out to be higher or lower based on images in the wild.
There are good arguments against this system but most of the technical debate seems to have devolved into amplifying lies now.
Still. Why trust them after that?
If a company can make my own smartphone report me to the police, and they want my business, they better prove I can trust them. Apple has plainly done the opposite.
The whole ordeal is just utterly 1984.
In contrast, a NH system believed to have a collision chance of 1 in a trillion trillion may well be considered infallible, and any detection be directly reported as CSAM, with the 'backend verification' amounting to nothing more than a rubber stamp.
Of course, if you implicitly trust Apple not to do the second, than you're right, the NH collision rate doesn't matter too much.
Are they?
Issues in any other Apple software will not send the police on you. Why would you install a software on your desktop/laptop that is designed to snitch on you, you would need to get some advantage or be forced by some law.
For now I see only disadvantages but please let me know of any real advantage and not speculation
Disadvantages:
- closed software with hidden db can't be trusted, so as a user you will always have a doubt that some non CP images are in the db(Apple always collaborates with governments)
- bugs in this stuff will cause you big problems(we seen in the past how false accusation destroyed peoples life) and we also seen bad actors abusing this kind of stuff.
- this is also clearly a beginning, now that Apple has the capability then even if they were saints a judge could force them to add new hashes, change the configs etc.
2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.
Now, I don’t trust them anymore.
Now apple is getting a local client side scanning tool ready. Interesting timing.
It could also happen that you lose your phone and "a human" finds it and randomly puts in the correct passcode on the first try and visually inspects your personal images. In fact, that seems vastly more likely [1].
[1] About 4% of smartphones are lost or stolen every year [https://www.mcafee.com/blogs/consumer/family-safety/almost-5... ], but make it just 1/1000, so 1e-3. Then a 6 digit passcode, 1e-6, so we're at 1e-9 per year, or 1000x as likely as being falsely flagged, assuming Apple's numbers (which can easily be achieved by calibrating the threshold).
0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever.
1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image.
2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No.
First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens.
Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs).
The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security.
Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly".
3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not.
[1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag...
[2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle
[3] https://news.ycombinator.com/item?id=28236102
[4] https://security.googleblog.com/2017/02/announcing-first-sha...
[5] https://www.zdnet.com/article/sha-1-collision-attacks-are-no...
Let's set aside the questions of where you got all these hashes to generate collisions with, how you got 30 of these mangled images into your victim's camera roll without them noticing. And let's also set aside whether your victim's device is an iPhone with iCloud Photo Library enabled (and has sufficient storage). I still don't get what these mangled images have achieved, other than giving the manual review team something other than child porn to look at.
Seems to me like it'd be easier to just find actual child porn, print it out, place it somewhere in the victim's house and then report it to the police.
It's inconceivable that anyone could desire possession of NCMEC-catalogued CSAM images without being aware that they're risking serious consequences if they're caught. Who wants their deepest, darkest, potentially life-ruining secrets just milling about with photos of the dog and last night's dinner?
[1] ...which is all but impossible for an average user to prove was effective; it's not like the Photos app has a "Not Child Porn!" checkmark.
But if nobody would import CSAM into their icloud library why do all the pictures need to be scanned in the first place? I would imagine anybody doing major illegal stuff being informed about important measures in order to not be caught.
Perhaps it's a prerequisite for deploying end-to-end encryption of iCloud Photo Library and/or iCloud Backups. The latter in particular has remained decryptable by Apple supposedly due to pressure from the FBI. Perhaps CSAM is what the FBI are using to justify their pressure.
Perhaps it's because Apple's team of lobbyists are seeing ahead to future anti-privacy, anti-encryption legislation being justified under the guise of CSAM. If Apple can show that the CSAM problem is already "solved" then such justifications disintegrate.
1. Identify some innocuous pictures that many many people have (memes, Beyoncé, whatever).
2. Produce CSAM.
3. Mangle it such that it is still CSAM visually, but NeuralHash-collides with the innocuous pictures from step 1.
4. Distribute.
5. Wait until they are (via some other mechanism) a) identified as CSAM, b) added to the NCMEC database, c) added to the Apple on-device database of blinded hashes in some iOS update.
6. Millions of people are suddenly incorrectly flagged for exceeding the threshold by NeuralHash (since they have the innocuous pictures in their library), and the review teams are flooded and can't pick out the small number of actual CSAM holders.
That is not without a certain elegance. However, it seems to me that
A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.
B) it would be quickly defeated by amending the 2nd tier algorithm (between NeuralHash and human review), though, as you highlight, that might be tricky given that the team working on this presumably only has access to the innocuous false positive collision image, not the (purposefully mangled) CSAM.
Note that this requires no single "desired" target picture. There are millions of popular, innocuous pictures. As long as you can make your CSAM match any one of them without significant mangling, you're good to go. Not having to choose one specific target makes this much easier to accomplish.
You can. Here is an example I created (with links to more): https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
I'm so tired of people suggesting that you can't. Please explain to me why you posted suggesting otherwise.
I've contemplated making some that are also photodna matches, I expect that it's possible. But access to photodna is only through some awful windows tools, and AFAICT people would just keep posting denials even after an example was posted-- so it's not worth the effort at least not worth it just to further the public discussion.
a) I didn't suggest otherwise, I said that it is predicated on that assumption, about which I was undecided, largely because b) I didn't know better.
I read that thread 7 days ago, when the collisions were a gray blob or a clearly modified dog (to Lena) or clearly modified Lena (to dog). I hadn't re-read the thread in the last 4 to 5 days, when you demonstrated the natural looking collisions (second-preimage images).
Very impressive work I wasn't aware of.
That’s a really interesting attack vector I hadn’t seen mentioned previously.
Most people are talking about the potential for adversarial images to be sent to users. If they were instead injected into the database itself (either by poisoning real CSAM or social engineering) that would have far wider ramifications.
I wonder what the most widely-saved pornographic images are across iCloud users.
If actual CSAM were perturbed to match the hash of, say, images from the celebrity nude leak a few years back and added to the database then thousands of users could be sent to “human review”. Since the images are actually explicit how would the human reviewers know not to flag them to authorities?
People don't seem to grasp what kind of images end up in the CSAM databases. They are most definitely not "leaked celebrity nude selfie" level stuff.
Think of the most vile sexual thing you could do to a child and then times that by two and halve the child's age in your mind. That's the shit that gets in there.
It's not something even 4chan weebaboos share. It's stuff that makes Liveleak regulars go "ewwwww, gross".
I think they can tell the difference between a naked celebrity and a molested child.
(1) Review by apple staff (2) Access and leaking by other apple staff (3) Access by hackers who have compromised their system (4) Access by parties coercing apple/staff, including via national security letters.
All of which compromise the privacy of the user. This matters or the neuralhash comparison wouldn't exist in the first place.
Totally agree that the whole system is grotesque-- but that doesn't stop it also being grotesque in every detail as well. The fact that there are false positives when they easily could have designed a system that had none (at the expense of increased false negatives) shows that Apple doesn't especially value customer privacy even if you accept their vigilante privacy invasion. The fact that it's possible to construct adversarial false positives and that their reports didn't disclose this fact shows they either don't know what they're doing or they're not being honest about the risks (or both).
The system is specifically designed so that colliding images does not pose a threat to the user.
NeuralHash and the CSAM scanning is grotesque, but please, criticize it for what it is, not some bullshit that is easily dismissed as technical ignorance.