News for hackers. Stuff that matters.
Re:Apple’s new abuse prevention system: an antritust/competi... by fossuser on Friday August 06, 2021 @07:56PM (#28094220)
This is wrong - the iCloud check is against known CSAM hashes, the false positive rate is essentially zero.
Re:Apple’s new abuse prevention system: an antritust/competi... by pyuser583 on Friday August 06, 2021 @09:35PM (#28094868)
This isn’t a normal file hash. It’s not SHA or bcrypt. A wide variety of states (1s and 0s) can have the same hash.

The idea is to take an image and have all of its possible derivatives create the same hash.

For example, if a hash was made of the Mona Lisa, any copy no matter how large, small, black and white, would have the same hash.

Think of all the ways the Mona Lisa could be transformed and still be the Mona Lisa.

The combinations of ones and zeros would be in the billions. If not more.

And all those possible combinations of ones and zeros go back to the same “hash.”

That’s extremely resourceful intensive.

My guess is that they are going to transform the images into a very low resolution, black and white thumbnail. Then compare it against known abuse images that have been similarly transformed.

Or they’re using AI. They might be using AI.

Either way, it’s guesswork. How many images might be transformable to the same black and white thumbnail. I don’t know.

Re:Apple’s new abuse prevention system: an antritust/competi... by bnj on Friday August 06, 2021 @09:11PM (#28094729)
Yes but my understanding is that the hashes are perceptual, as opposed to cryptographic.

If the system was matching against known cryptographic hashes the collision / false positive rate would be small, but the fuzzy matching involved with perceptual hashing necessarily has a greater false positive rate.

And that doesn’t even begin to address the detection of sent and received “explicit images” which are detected on device and don’t have a set of known hashes.

Re:Apple’s new abuse prevention system: an antritust/competi... by fossuser on Friday August 06, 2021 @09:14PM (#28094757)
I suspect that's why they have some threshold that moves the false positive rate to one in one trillion.

The iMessage bit is different - it's only on device, only on child accounts, and only alerts parents. It's more akin to a parental control feature than anything else.

Re:Apple’s new abuse prevention system: an antritust/competi... by samatman on Saturday August 07, 2021 @12:17AM (#28095646)
It was pretty dumb of Apple to announce both of these things at the same time.

They have nothing to do with each other, and I've seen a dozen people on HN confuse them. If the message is getting muddled here, it will be hopelessly conflated in less technical circles.

I'm concerned and upset about the CSAM filter for all the reasons that keep hitting the front page, but don't care about the opt-in parental controls at all, and if I had kids, I might want them.

But if I thought the CSAM filter worked like the nudie-detector filter, I'd be wigging out.

Re:Apple’s new abuse prevention system: an antritust/competi... by skinkestek on Saturday August 07, 2021 @12:44PM (#28099927)
> I suspect that's why they have some threshold that moves the false positive rate to one in one trillion.

So now instead of sending just one nice innocent very high resolution images of "Tokyo City" or something with something horrific hidden somewhere you have to send a few such images.

That is reassuring. I can never believe anyone except me will think about that.

(If the system is too dumb to detect this it is worthless, and if it is smart enough this opens the floodgates for anyone wanting to make trouble for just about anyone.)

Re:Apple’s new abuse prevention system: an antritust/competi... by nomel on Sunday August 08, 2021 @03:56AM (#28104982)
This would require multiple hash collisions.
Re:Apple’s new abuse prevention system: an antritust/competi... by skinkestek on Monday August 09, 2021 @07:38AM (#28115453)
No need for hash collisions, that's what makes it so bad: just edit multiple known verybad images in multiple nice high res photos like I tried to explain above.
Re:Apple’s new abuse prevention system: an antritust/competi... by nomel on Monday August 09, 2021 @12:22PM (#28118800)
It’s a visual hash, not something like an md5. And now you’re talking of a directed attack to intentionally make hash collisions?
Re:Apple’s new abuse prevention system: an antritust/competi... by lamontcg on Saturday August 07, 2021 @02:21AM (#28096236)
It doesn't matter if the cryptographic algorithm is one in a trillion. If there's a concurrency bug in the application which applies the algorithm the wrong account could be flagged against a legitimate image in the database. If the human involved overly trusts the system they may not validate against the actual file in the users account, or may assume the user deleted it somehow and figure its "safter" to let law enforcement figure it out.

Bugs in the application of the code, combined with human complacency and mistakes can certainly lead to errors, even if the cryptographic algorithm itself was perfect.

We really need to bring back comp.risks

Re:Apple’s new abuse prevention system: an antritust/competi... by bnj on Friday August 06, 2021 @10:47PM (#28095251)
This is a great point, thanks for adding that detail.
Re:Apple’s new abuse prevention system: an antritust/competi... by paul_f on Friday August 06, 2021 @07:59PM (#28094243)
This seems naive, there are always false positives
Re:Apple’s new abuse prevention system: an antritust/competi... by fossuser on Friday August 06, 2021 @08:00PM (#28094252)
It’s not naive, it’s math.
Re:Apple’s new abuse prevention system: an antritust/competi... by mpol on Friday August 06, 2021 @08:19PM (#28094376)
It's not a math problem, it's a human problem.

suppose you have a partner who is a 'petite' woman of 34. She enjoys posting nudies on a website, but without her face in that picture. Someone who collects child porn downloads it, because he enjoys that picture. A year later he gets caught by the police and all his pictures get marked as 'verified child porn'. Suddenly you get marked as owning child porn.

Re:Apple’s new abuse prevention system: an antritust/competi... by fossuser on Friday August 06, 2021 @08:25PM (#28094415)
That isn’t CSAM.

https://www.nytimes.com/interactive/2019/09/28/us/child-sex-...

Apple’s thing has some sort of threshold anyway so one image would not trigger it. I don’t buy your example - the CSAM images are not what you’re describing.

Re:Apple’s new abuse prevention system: an antritust/competi... by farmerstan on Friday August 06, 2021 @11:03PM (#28095312)
Who determines whether something is CSAM? How do you or I know that every single one of those images actually is CSAM? How do we know if the FBI or CIA or CCP adds hashes of innocent pics in order to pin a crime against someone?
Re:Apple’s new abuse prevention system: an antritust/competi... by DSingularity on Friday August 06, 2021 @08:37PM (#28094486)
No. You can design a system where the FPR is essentially zero. Even if shitty md5 is used.
Re:Apple’s new abuse prevention system: an antritust/competi... by Conlectus on Friday August 06, 2021 @08:51PM (#28094587)
Perceptual hashes are not exact hashes, otherwise they would be useless for this task; you would just mirror the image or change 1 pixel.

They are instead fuzzy classifiers, and thus have non-zero error rates.

Re:Apple’s new abuse prevention system: an antritust/competi... by nomel on Friday August 06, 2021 @08:59PM (#28094656)
But this trigger requires many of these false flags to exceed the threshold (most likely why it exists). I imagine the "one in a trillion" numbers they claim for false flagging rate are probably cemented in reality, and make it trivial for human review.
Re:Apple’s new abuse prevention system: an antritust/competi... by skinkestek on Saturday August 07, 2021 @12:48PM (#28099957)
I've explained the problem here: https://news.ycombinator.com/item?id=28099927
Re:Apple’s new abuse prevention system: an antritust/competi... by DSingularity on Friday August 06, 2021 @11:26PM (#28095420)
How does this relate to the allegation I was responding to? There is a difference between false positives and false negatives you know. False negative is where the criminal evades the system through measures you describe. False positive — what I was responding to — is where Apple incorrectly accuses an innocent person and thus violates their privacy during the subsequent review.
Re:Apple’s new abuse prevention system: an antritust/competi... by Zababa on Friday August 06, 2021 @09:07PM (#28094701)
> No. You can design a system where the FPR is essentially zero. Even if shitty md5 is used.

How can you do that, considering md5 can have collisions?

Re:Apple’s new abuse prevention system: an antritust/competi... by schoen on Friday August 06, 2021 @10:47PM (#28095244)
A confusing thing that I think people haven't addressed clearly for the most part:

For a hash (whether cryptographic or perceptual), there is a chance of random collisions and also a difficulty factor for adversarially-created intentional collisions. The random collision probability has to be estimated based on some model of the input and output space (with cryptographic hash functions, you would usually model them as pseudorandom functions and assume that the collision probability is the same one created by the birthday paradox calculation).

Intentional collisions depend on insight about the structure of the hash function, and there are also different kinds of difficulty levels depending on the nature of the attack (preimage resistance, second-preimage resistance, and collision resistance). Gaining more insight about the structure of the hash function can act to reduce the work factor required for mounting these attacks. That should be true for perceptual hashes just as much as cryptographic hashes, but presumably all of the intentional attacks should start off easier because the perceptual hashes' threat models are weaker and there's much less mathematical research on how to achieve them.

And in AI systems involving classifiers, it was generally easy for people to create adversarial examples given access to the model. Perceptual hashes for estimating similarity to specific known images aren't the exact same thing because it's less like "how much like a cat is this image?" and more like "how much like NCMEC corpus image 77 is this image?", but maybe some of the same techniques would still work. In the cryptographic hash analogy, I guess that would be like trying to break preimage resistance.

Re:Apple’s new abuse prevention system: an antritust/competi... by Zababa on Friday August 06, 2021 @11:22PM (#28095398)
Thanks for the detailed explanation. I understand why that works for perceptual hashes if you make them really precise, however I doubt it would work with md5, which is why I asked.
Re:Apple’s new abuse prevention system: an antritust/competi... by DSingularity on Friday August 06, 2021 @11:35PM (#28095459)
The discussion I thought we were having was about false positives and not adversarially induced false positives. For the former the random collisions have a probability of 1/(2^64).

To mitigate adversarial false positives one idea is to use the combination of a cryptographically strong hash along with a randomly selected perturbation of the file. Prior to hashing, perturb the file and submit both the hash and the selected perturbation to apple. Apple selects the DB based on the perturbation and proceeds with matching and thresholding.

Re:Apple’s new abuse prevention system: an antritust/competi... by skinkestek on Saturday August 07, 2021 @12:50PM (#28099980)
> The discussion I thought we were having was about false positives and not adversarially induced false positives.

I think the rest of us have been discussing how this can and will be abused, by definition by adversaries.

Many of us have also observed for years how systems are abused so we sadly have a gut feeling for this.

Re:Apple’s new abuse prevention system: an antritust/competi... by Zababa on Saturday August 07, 2021 @12:20AM (#28095659)
I thought that the random collision for md5 was way higher than that. If it's that low, you're right, this would work. I'm not sure I understand the part about the pertubation.
Re:Apple’s new abuse prevention system: an antritust/competi... by DSingularity on Saturday August 07, 2021 @10:07AM (#28098691)
It’s actually 1/(2^128).

If the attacker does not know how the image will be perturbed prior to hashing then he cannot generate an image which matches with known CSAM.

Re:Apple’s new abuse prevention system: an antritust/competi... by hhh on Friday August 06, 2021 @07:58PM (#28094240)
One in one trillion chance per year, per the paper on the Apple site.
Re:Apple’s new abuse prevention system: an antritust/competi... by patmcc on Friday August 06, 2021 @09:12PM (#28094740)
I see the 'one in one trillion' but it's a bit vague; I read it as '1 in trillion chance' per image.

So when we have a billion iPhones in the wild taking 10 images a day...1 in a trillion chances happen every few months. Now, if that triggers some further review, maybe that's an acceptable false positive. If it triggers a SWAT team, I don't think it is.

Re:Apple’s new abuse prevention system: an antritust/competi... by jeromegv on Saturday August 07, 2021 @10:01AM (#28098635)
They have also indicated that a single match won't be enough to trigger it, so an accidental match (being that 1 in a trillion person) is not enough.
Re:Apple’s new abuse prevention system: an antritust/competi... by fossuser on Friday August 06, 2021 @08:00PM (#28094250)
Yeah, that’s essentially zero.
Re:Apple’s new abuse prevention system: an antritust/competi... by jhayward on Friday August 06, 2021 @09:52PM (#28094957)
I would like to bet Apple's market value against that number being correct in an adversarial context.
Re:Apple’s new abuse prevention system: an antritust/competi... by 2pEXgD0fZ5cF on Friday August 06, 2021 @08:01PM (#28094259)
> the false positive rate is essentially zero

I highly doubt that

Re:Apple’s new abuse prevention system: an antritust/competi... by DSingularity on Friday August 06, 2021 @08:38PM (#28094492)
Based on what? Unless you point out a flaw in the math, it’s zero.
Re:Apple’s new abuse prevention system: an antritust/competi... by 2pEXgD0fZ5cF on Friday August 06, 2021 @09:05PM (#28094686)
> Based on what?

Based on the fact that ultimately we can't check the system. And based on the fact that at some point in the chain humans are involved. [1]

What we are left with is "trust in Apple" not "trust in math".

[1]: https://news.ycombinator.com/item?id=27878333