[0] https://www.dictionary.com/e/what-is-stochastic-terrorism/
[1] https://www.merriam-webster.com/words-at-play/history-of-the...
How did we get here? How did everything become so politicized and polarized? How did law enforcement become so militarized? How did we as a society become so terrified and distrustful of our neighbours?
Not true. Hash matches are to be human reviewed. So no, people won't get "swatted" accidentally as you allege.
The other concerns people have been voicing are certainly valid though (IMHO).
Until it proves too expensive, then a different AI system will do it instead. I have zero faith that it'll be a fully competent, well trained, well rested, well paid person will actually be doing these reviews in the long run.
I mean, seriously, you're assuming as beyond obvious something that's demonstrably wrong RIGHT NOW.
So no, under the proposed implementation (and current judicial requirements in the west) robots dispatching armed police to kill pets or people without ANY human oversight because of a situation that is not time sensitive is mere fantasy.
Don’t misunderstand, I am against this whole thing for the reasons many others have well articulated in this thread.
But the claim that robots are dispatching armed police without any human interference RIGHT NOW is provably wrong.
Show me one incident where this has happened already and I will donate $100 to the charity of your choice.
And yes, the judicial rules and Apples implementation could change in the future. That’s certainly a risk but is not the case RIGHT NOW. I mean, seriously?
Especially since anything flagged by this system is manually reviewed by Apple. So, there would exist counter-evidence for the govt claim.
Don’t misunderstand, I see how this system is ripe for abuse. I was just commenting on the specific claim that there will be automated SWAT call outs (presumably in the US).
The problem is that they can do whatever if you have CP. Emphasis on "have": you might not even know you have it, because all it takes for you to be guilty is for a forbidden bit of data to be on your disk or cloud account. How did it end there? Doesn't matter much. It's unlikely you'll be convicted if someone else maliciously puts CP on your drive, but it's extremely likely your whole neighborhood will know about it before it's settled. Think about that.
> They don’t need Apples system.
Apple's system makes it sure that whenever someone puts CP on someone else's account, it rings the police and starts the nightmare. Without it, there's a few extra steps that make the nightmare much less likely to happen in the first place.
The only argument I made was against the claim that fully automated police dispatches will occur because of this system (in the west).
Making outrageous, tinfoil hat level claims does not help our collective argument against this system.
It just makes us look like paranoid conspiracy theorists.
Let’s make coherent arguments (like the one you made above) instead of fantasy ones.
That said, I don’t like my phone being a snitch.
This comment suggests this phrase might be cleverly worded, to make it seem like the images are human reviewed, while that actually not being the case: https://news.ycombinator.com/item?id=28096059
The idea is to take an image and have all of its possible derivatives create the same hash.
For example, if a hash was made of the Mona Lisa, any copy no matter how large, small, black and white, would have the same hash.
Think of all the ways the Mona Lisa could be transformed and still be the Mona Lisa.
The combinations of ones and zeros would be in the billions. If not more.
And all those possible combinations of ones and zeros go back to the same “hash.”
That’s extremely resourceful intensive.
My guess is that they are going to transform the images into a very low resolution, black and white thumbnail. Then compare it against known abuse images that have been similarly transformed.
Or they’re using AI. They might be using AI.
Either way, it’s guesswork. How many images might be transformable to the same black and white thumbnail. I don’t know.
If the system was matching against known cryptographic hashes the collision / false positive rate would be small, but the fuzzy matching involved with perceptual hashing necessarily has a greater false positive rate.
And that doesn’t even begin to address the detection of sent and received “explicit images” which are detected on device and don’t have a set of known hashes.
The iMessage bit is different - it's only on device, only on child accounts, and only alerts parents. It's more akin to a parental control feature than anything else.
They have nothing to do with each other, and I've seen a dozen people on HN confuse them. If the message is getting muddled here, it will be hopelessly conflated in less technical circles.
I'm concerned and upset about the CSAM filter for all the reasons that keep hitting the front page, but don't care about the opt-in parental controls at all, and if I had kids, I might want them.
But if I thought the CSAM filter worked like the nudie-detector filter, I'd be wigging out.
So now instead of sending just one nice innocent very high resolution images of "Tokyo City" or something with something horrific hidden somewhere you have to send a few such images.
That is reassuring. I can never believe anyone except me will think about that.
(If the system is too dumb to detect this it is worthless, and if it is smart enough this opens the floodgates for anyone wanting to make trouble for just about anyone.)
Bugs in the application of the code, combined with human complacency and mistakes can certainly lead to errors, even if the cryptographic algorithm itself was perfect.
We really need to bring back comp.risks
suppose you have a partner who is a 'petite' woman of 34. She enjoys posting nudies on a website, but without her face in that picture. Someone who collects child porn downloads it, because he enjoys that picture. A year later he gets caught by the police and all his pictures get marked as 'verified child porn'. Suddenly you get marked as owning child porn.
https://www.nytimes.com/interactive/2019/09/28/us/child-sex-...
Apple’s thing has some sort of threshold anyway so one image would not trigger it. I don’t buy your example - the CSAM images are not what you’re describing.
They are instead fuzzy classifiers, and thus have non-zero error rates.
How can you do that, considering md5 can have collisions?
For a hash (whether cryptographic or perceptual), there is a chance of random collisions and also a difficulty factor for adversarially-created intentional collisions. The random collision probability has to be estimated based on some model of the input and output space (with cryptographic hash functions, you would usually model them as pseudorandom functions and assume that the collision probability is the same one created by the birthday paradox calculation).
Intentional collisions depend on insight about the structure of the hash function, and there are also different kinds of difficulty levels depending on the nature of the attack (preimage resistance, second-preimage resistance, and collision resistance). Gaining more insight about the structure of the hash function can act to reduce the work factor required for mounting these attacks. That should be true for perceptual hashes just as much as cryptographic hashes, but presumably all of the intentional attacks should start off easier because the perceptual hashes' threat models are weaker and there's much less mathematical research on how to achieve them.
And in AI systems involving classifiers, it was generally easy for people to create adversarial examples given access to the model. Perceptual hashes for estimating similarity to specific known images aren't the exact same thing because it's less like "how much like a cat is this image?" and more like "how much like NCMEC corpus image 77 is this image?", but maybe some of the same techniques would still work. In the cryptographic hash analogy, I guess that would be like trying to break preimage resistance.
To mitigate adversarial false positives one idea is to use the combination of a cryptographically strong hash along with a randomly selected perturbation of the file. Prior to hashing, perturb the file and submit both the hash and the selected perturbation to apple. Apple selects the DB based on the perturbation and proceeds with matching and thresholding.
I think the rest of us have been discussing how this can and will be abused, by definition by adversaries.
Many of us have also observed for years how systems are abused so we sadly have a gut feeling for this.
If the attacker does not know how the image will be perturbed prior to hashing then he cannot generate an image which matches with known CSAM.
So when we have a billion iPhones in the wild taking 10 images a day...1 in a trillion chances happen every few months. Now, if that triggers some further review, maybe that's an acceptable false positive. If it triggers a SWAT team, I don't think it is.
I highly doubt that
Based on the fact that ultimately we can't check the system. And based on the fact that at some point in the chain humans are involved. [1]
What we are left with is "trust in Apple" not "trust in math".
Problem is that this scanning is necessarily fuzzy and there is going to be a false positive rate to it. And the way that you'll find out that you've tripped a false positive is that the SWAT team will knock your door down and kill your dog (at a minimum). Then you'll be stuck in a Kafkaesque nightmare trying to prove your innocence where you've been accused by a quasi Governmental agency that hides its methods so the "bad guys" can't work around them.
It isn't just "authoritarian regimes" abusing it, it is the stochastic domestic terrorism that our own government currently carries out against its own citizens every time there's a beaurocratic fuckup in how it manages its monopoly on violence.
This is the "Apple/Google cancelled my account and I don't know why" problem combined with SWATing.