There has been a lot of hyperbole going around and the original premise that this is a breach of privacy is still true, but in my opinion the actual repercussions of attacks and collisions are being grossly exaggerated. One would have to create a collision with known CSAM for both algorithms (one of which is secret) which also overlaps with a legal porn image that could be misconstrued as CSAM by a human reviewer, or at the very least create and distribute hundreds of double collisions to DOS the reviewers.
They are exactly the same case.
A cryptographic key is a piece of information which, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system. This means that your system should remain secure even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs). If the key fails to have this property, your cryptosystem is broken.
The trained model (or the weights of a NN) does not have this property at all. Keeping the model secret does not ensure the confidentiality or integrity of the system. E.g. just knowing some inputs and outputs of the secret model allows you to train your own classifier which behaves similarly enough to let you find perceptual hash collisions. If you treat your model as a cryptosystem, this would be a known-plaintext attack: any system vulnerable to these is considered completely and utterly broken.
You'd have to keep all of the following secret: the model, all its inputs, all its outputs. If you manage to do that, this might be secure. Might. But probably not. See also Part 2 of my FAQ, which happens to cover this question. [1]
This seems highly unlikely. You could train a model to find those exact known hashes, but I highly doubt you could get it to accurately find any other unknown hash.
> You'd have to keep all of the following secret: the model, all its inputs, all its outputs.
These are all, in fact, secret.
Your "highly doubt" is baseless. Black box attacks (where you create adversarial examples only using some inputs and outputs, but not the model) on machine learning models are not new. They have been demonstrated countless times [1]. You don't need to know the network at all.
> These are all, in fact, secret.
This is not the case, since regular, unprivileged Apple employees can and will look at the inputs and outputs of the model (the visual derivatives and their hashes). It's also irrelevant.
You insist that there is some kind of analogy between "keeping the model secret" and keeping a "cryptographic key" secret. There is no such analogy. It makes no sense. It is simply not there. Keeping a cryptographic key secret keeps confidentiality and integrity. Keeping your model secret accomplishes neither of these.
[1] https://towardsdatascience.com/adversarial-attacks-in-machin...
This is not a machine learning model as such, though, and is used differently than they are.
> This is not the case, since regular, unprivileged Apple employees can and will look at the inputs and outputs of the model
Can they?
But then they still need to upload the original image to the server, and what was the reason for doing the scanning client-side then when they still upload it?
Probably so that China cannot force Apple to hand over arbitrary images in iCloud (or all images in iCloud). With Apple's design the only images China can get from you are malicious images that people send you. If Apple scanned every image serverside without any clientside scanning, then theoretically China could get all newly-uploaded iCloud images.
They also keep the second hash function private.
Again, really naive but it seems like if you have two distinct multi-dimensional hashes it would be much harder to solve the gradient descent problem.