I would hope most privacy conscious people disable iCloud, but that’s another story.
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
To be clear each image, the image’s NeuralHash, and a visual derivative are uploaded to iPhoto. This allows for the inspection of the NeuralHash algorithm used which I actually prefer.
The phone isn’t downloading the hash database.
You can setup secure encrypted backups, but the customer losing the key means losing the back so that’s not what consumer focused companies are going to do. In other words any backup service that doesn’t have big warnings that losing your key loses your backup means they can read your data.
"The mud puddle test: You don’t have to dig through Apple’s ToS to determine how they store their encryption keys. There’s a much simpler approach that I call the ‘mud puddle test’"
They did add a classifier to iMessage. But it's designed to prevent children seeing any sexually explicit images.[2] There wouldn't be a reason to train it on images of children specifically.
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[2] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...
Perceptual matching is used to sort categories of images. A quick DuckDuckGo will turn up many results. No stretch of the imagination will turn this into a bit for a bit comparison. This is a machine learning algorithm used to categorize images. https://www.ibm.com/blogs/research/2019/10/learning-implicit...
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hashing”
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
“Perceptual hashing is the use of an algorithm that produces a snippet or fingerprint of various forms of multimedia.[1][2] A perceptual hash is a type of locality-sensitive hash, which is analogous if features of the multimedia are similar.”
https://en.wikipedia.org/wiki/Perceptual_hashing
The goal is to verify a black and white copy of an image is identical to a colored original. Search algorithms want a similar thing so they can validate an image contains a blue car. However, a perceptual hashing algorithm must differentiate between different images containing a blue car while matching a photoshopped copy of the same image.
Citation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
The perceptual hashing is based on AI techniques. “The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process.”
The difference is AI classification is based on defining something as say a “Cat” and then the AI spits some association with how cat like the image is. This extracts features from an image then compares lists of features to specific images.
From the PDF:
"The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements.
The neural network that generates the descriptor is trained through a self-supervised training scheme. Images are perturbed with transformations that keep them perceptually identical to the original, creating an original/perturbed pair. The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. "
Image classification on the other hand cares about if the image contains say a stop sign or a trash can. That’s useful for self driving cars etc.
Aka classification you might want to match two different bands playing the same song as identical. Where perception hashing would want them to be classified differently.
It is a perceptual Hash of the images characteristics and perceived continent by the algorithm, that’s the “neural” in neuralmatch.
Apple, for yours has been building-in machine learning dedicated chips into their builds so this shouldn’t affect battery life.
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hasing"
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
Again, Apple nor any company, have access to the source data, just hashes.
People can argue against this approach for privacy reasons, but I think the false positive argument is a relatively weak one.
There will be many false positives, they will be reviewed by people. When there’s more than a few false positives, you will be investigated by the FBI.
It's fuzzy hash, not ""AI"", based. Cloudflare uses it too, and last time i checked the web is still functional.
https://support.cloudflare.com/hc/en-us/articles/36004610611...
How about we turn the tables and have the complainers suggest a solution. Because every single time an approach to targeted child porn takedowns has been suggested, such as datacenter raids which would not affect as many people, someone is screaming about their privacy.
Child abusers evolve and are very happy if law enforcement doesn't.
Being worried about privacy, establishing a precedent for scanning my data against a government database, and the risk of false positives with such an insanely emotionally charged crime is more than mere complaining.
The onus should not be on me to justify why this shouldn't be done. This is something new and it is perfectly fine to argue against it without needing to provide an alternative.
That being said, my solution is to continue to follow the process that law enforcement is currently using.
If you knew how they approach this you wouldn't be satisfied either
Yes. And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system. The only effect this will have is that moral panickers like you will have got everybody's privacy invaded over your moral panic of the day.
They don't use simple file hashes to match images, but perceptual hashes. That way they can find modified derivatives of a source image. The problem with this approach, though, is that this is ripe for false positives. Two completely unrelated images can have similar hashes.
This is, and always is, a game of cat and mouse. Law enforcement is always catching up. They are the cryptologists here. They are never ahead, always behind, because they don't know the new protections peddlers are using until they have been in use and later discovered.
No matter what vector you plug, they will use another, and the game continues (sick game). Maybe divide the image into 32 different quadrants and rearrange them, then put them back in the correct order when viewing through a specific image viewer. I'm sure that would bypass whatever detections they've come up in their fuzzy fingerprinting with as the entire image is now different. By the time they catch someone using this, they'll have already moved on to something different, as they always do.
I will never be ok with warrantless searches of my personal property, no matter the reason or justification or subject, and no matter who it is done by (government or private company). And I say that as a survivor of some pretty horrific shit as a kid to the point I fucking tremble with absolute rage when thinking about it 35+ years later. I would be banned from everything for life if I were to honestly state what I would do with these types of people. The movie "Saw" is tame in comparison. I have no compassion or sympathy for these sickos. But when reading world history, I can absolutely see the importance of "innocent until proven guilty" and Blackstone's Ratio "It is better that ten guilty persons escape than that one innocent suffer." Most of human history was the opposite, and it was brutal and full of literal witch hunts. Are we progressing as a species, or regressing in terms of human rights when it comes to technology?
The specific subject of child abuse is irrelevant in my commentary. It was a commentary on the general category of AI, used all over, for many things, and more and more every day, but nice try.
Edit: You edited out what I was referring to as I was replying.
If they're using fuzzy matching with perceptual hashes, then the space that false positives can exist in for each perceptual hash is huge.
The definition of child porn varies around the world. These systems use the US definition. This is not entirely what you might expect. For example, in the USA the courts have decided that cartoons can be child porn even though no actual children are in the picture. Most of the world does not agree with this, meaning an image can be CP in one place but not another. Is Apple going to enforce the US definitions or the ones where the user actually lives?
In the USA, photos an under-age person takes of themselves can also be considered CP.
What counts as a "child" for sexual purposes also varies around the world. Some countries have a lower age of consent than other places. In some parts of the world the age of consent and the age at which a child stops being a child for CP purposes are different, meaning that a teenager can have sex legally but if they take a photo of themselves doing it, they are trafficking in CP.
Finally, what is actually on these image blacklists? Hardly anyone actually knows because of the third rail nature of CP. Tech firms are often delivered image hashes, not even the images themselves, by third party 'charities' of various kinds and tech workers are - for obvious reasons - not normally given access to the actual pixels. Additionally, appeals from users are invariably ignored because people say "legal issues, it's complicated" and so everyone clams up. If FPs occur there is no way to resolve it and the people who see your appeal, if there even is one, won't be willing to actually look at the image to find out what it was.
It should be obvious how much potential for abuse this hands the people who actually manage these CP databases. Literally any image can be made verboten immediately, without any recourse, and basically nobody will ever find out including the people who shut down the affected users.
Fuzzy means that it takes compression and the like into account, because even if just one pixel out of 20 thousand is different, the hash is different too. Fuzzy hash still recognizes it as the same image, so using an algorithm to alter the color etc. won't work.
That's also true for the no-fly list and the Terrorist Screening Database,[1] yet those are full of false positives. And unlike those lists, CSAM databases cannot be independently verified. To do so would require having the original images, which is illegal.
1. https://en.wikipedia.org/wiki/Terrorist_Screening_Database
So if you're charged on the basis of a fuzzy hash matching, you'd subpoena Apple for the photo in your backup that matched, present it to the court (since it doesn't actually matter if it's CP or not to be admissible), and you win the case.
0. https://www.johntfloyd.com/the-difficulty-with-criminal-evid...
No. NeuralMatch was “trained” using 200,000 CP images. “Neural“ is likely a reference to the perceptual matching that it uses. It is not a bit for bit match.
Perceptual matching is a technique used for categorizing images based on characteristics and content.
The algorithm will scan your library containing new information and compare it to what it understands as CP.
They are using a library of human verified images to compare the hashes. Probably similar to PhotoDNA[0]. There can be false positives, but AFAIK the algorithms are not trying to identify naked children, but comparing 2 similar image hashes.
I didn't get that from the article, could you point to a source on this?
0: https://www.missingkids.org/theissues/csam#:~:text=Electroni... (ctrl-f "1,400")
So even if not running hashes against known porn, it could still be doing facial recognition. Wonder what kind of implications that would have for people in witness protection or battered spouses and other legitimately "disappeared" people.
“Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.”
Also from the article.
‘Trained’ being important.
Thank God, I don't use Apple.
(I should probably delete them, since I've basically never used them... although they could be useful for one of those end-of-world Raspberry Pi builds: https://back7.co/home/raspberry-pi-quick-kit-one Edit: the non-bad ones, in case this gets misconstrued)
Edit: I've deleted my archive.
Hoarding porn is kind of useless nowadays, especially without any kind of selection and categorization, but maybe someone does it because it's their hobby (or obsession), who knows.
Also if a lot of people has a bunch of porn images scraped from forums it's quite possible that eventually some of that might become illegal as the laws change here or there around the world. Should they proactively delete everything? Sure, they could. To be safe.
But this kind of safety is absolutely the bad kind on the long term. (Though of course it's possible to advocate for reform while not storing content with uncertain legality.)
I mean, that is actually illegal. The problem here is the possession of child pornography.
But, yeah, I've deleted them now.
And what, in this context, is "your data?" Is a photo on an iPhone Apple's data?
I'd argue that the thing you should do if you get a hit is to delete your data, unless you want to spend enormous amounts of time and money defending yourself from prosecution by people who believe that false positives are vanishingly rare.
I really don't care about the semantics of data ownership. If Apple wants to scan photos you upload to iCloud so they don't run into a scandal years down the road that "iCloud is being used to distribute CP" then that's their call.
And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
No FBI is called. Imagine the financial damage this would do if specialist come over and try to collect evidence and the company is forced to cooperated and work for them, for free ofc. Only the big tech does this. They have whole teams for this and automated system and what not because surely the FBI wont walk into googled data center ever to collect evidence. But all the smaller companies they cant afford to do this and the detection system is actually to protect them form going bankrupt because they accidentally stored some illegal data from a user.
>And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
Yes such laws exist. So they use filter software from third parties. They intentionally offer some generic filter that includes "unwanted content" and doesn't specify what it is when it find something so the companies can wave away the liability as they have no way to know if the image was valuable evidence of a crime or just some LiveLeak gore. They can justify that no human watches it because that's to expensive.
>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)
https://www.nbcnews.com/technolog/your-cloud-drive-really-pr...
There was another round of outrage when the Windows 10 TOS extended this to local storage.
It also has the use case of scanning for DRM violations. (ripped mp3 files or movies)
And the way it’s being done (hashes), a collision is highly unlikely. If it does occur it doesn’t mean it’s similar in nature (e.g. innocent picture of own child in bath). The hash isn’t looking at the image content in the sense of “what’s in the picture”, just the bits of the file. So it’s highly, highly unlikely, even if a collision occurs, that the collision would be an image that happens to be another child innocently bathing.
Actually, it's very easy to end up with an image that has a similar perceptual hash to an illegal image.
They are not doing MD5 hashing, they're taking perceptual hashes and then using something like the hamming distance or Levenshtein distance to make a fuzzy match with hashes from illegal images.
I've built products using these methods, and it is incredibly easy to make a fuzzy match based on perceptual hashes from two images that have nothing to do with each other.
Are you sure, given how many iPhone takeover/jailbreak bugs regularly exist, including the recent 0-click iMessage bug? Would you like to dare a hacking group, domestic or foreign, to land a single verboten image on your iPhone?
> And the way it’s being done (hashes), a collision is highly unlikely.
The limited details so far are suggestive that its using perceptual hashes, which are more susceptible to collision-engineering/false-positives than cryptographically-secure hashes.
Additionally, this isn't just a hash of the file but a perceptual hash on the image content. So e.g. changing a single bit in the image would create a different cryptographic hash, but generally not a different perceptual hash.
That's not good enough given how our media currently works. Imagine articles published when information about such a check leaks that say "celebrity X's phone checked by police for suspected CSAM." While that is the truth ("suspected") no one cares about that nuance and such a person would get cancelled very quickly, even if there was no evidence of wrongdoing.
Of course, there's no telling what images will be in the database when this inevitably expands beyond CP.
It's absurdly easy to change the hash of a file. In the case of something like JPEG, you don't even have to change the file itself, you can just change the metadata. Apple could presumably only hash the image itself, but again, all you have to do is make tiny, imperceptible to humans changes to the image and the hash is totally different.
Long story short, this is either nearly pointless and privacy invasive, or it's about to get drastically more invasive to be effective.
That's 115792089237316195423570985008687907853269984665640564039457584007913129639936. Probability of collision (two different inputs generating the same hash) is infinitesimal.
Edit: It makes sense that such a notification system would be impossible to implement. Perhaps, registered offenders should not only have their location placed on a map but their phones have certain privileges disabled or monitored.
The same does not go for random creeps wanting to take close up pictures of a child.
The creep behavior is the problem. The photograph is a symptom of it.
There are words like “neural” and “trained” here, but no clear sign that there is actually any kind of image recognition that would risk false positives other than hash collisions.
I am not sure the accuracy of these systems but even if they are pretty accurate there is something off putting about this trend.
This very much feels like a Guilty until proven innocent type of program... "if you have nothing to hide you have nothing to fear"...
Neither of those normally work out really well.
But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I agree though, the scary part might be how to clear your name after an overreaction occurs, similar to the effect swatting has on innocents or other kinds of false accusations. But those would happen even without such a system - we’ve heard about such stories before. “Man held for child porn charges, but it turned out they were innocent,” - it doesn’t take this kind of system to have police look over your phone for evidence and find a different crime than expected.
That all said… child porn is really, really terrible and I’m actually in favour of this. As a society, we can decide on what’s acceptable. As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes. I do agree though, all these systems might make it harder to catch perpetrators if it pushes them offline. But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I suppose the fundamental problem is that having more evidence doesn’t necessarily lead to justice if it paints a false picture.
This is a horrible idea, a massive regression in Apple (and therefore the broader industry's) stance on security/privacy, and the fact that you are dutifully swallowing the "child porn" justification is so incredibly concerning to me.
You're also being naive by swallowing the "it's only iCloud" justification. There is a clear intent to progress to scanning all local photos. After all, if photos must be match those of a government child abuse database, and any matches trigger manual review, what would an innocent person possibly have to hide? By that logic it'd be absurd to not scan just because it's only locally. After all, you're only catching bad guys! This is about protecting the children after all! /s
I oppose government-supplied blacklists. Doubly so when we can't even see what's in the database, but I would oppose them either way.
I don’t appreciate you calling me naive. There isn’t clear intent to scanning all local photos. In fact, it seems like the system was deliberately designed to only work with photos that are being uploaded to iCloud.
When someone is trying to rob you, generally the best thing to do is to hand over your wallet. If a law passes you should obey it for your own sake. That's an entirely orthogonal question to whether we as the users should be outraged or not.
Also while obeying an injust law, a company has a responsibility to legally challenge it in court with their resources, if for no other reason than to avoid the profit loss of having to comply with invasive regulations
Furthermore, it's not clear that this is the direct result of some federal law. It reads more like pro-actively appeasing their federal overlords.
It is good advice not to carry around huge amounts of money on your wallet. Someone might want it and have the power to get it.
Same with megacorps constructing and bragging about tools for oppression:
Better not do it.
Otherwise just be prepared that China will demand that you also upload their hashes of extremely bad photos - and one or two guys they want to get hold of.
It is kind of a Manhattan project for digital privacy - it is meant to bring a quick end to a actual real problem we have - but it has the potential to become a shadow over society for decades if not centuries to come.
This also limits the possible avenues for future exposure - if one has less ways to distribute or get CP or the overall amount of it is reduced, they may never get into it in the first place and the market for it will shrink. Thereby reducing the amount produced over time.
If in the US Do you like the concept behind 4th amendment? being free from unreasonable searches unless probable cause can be articulated that you have committed a crime?
It is very scary how effect the "think of the children" narrative is to get people to willing give up their liberty
I doubt they need, or want to match more than one. They'll use the reasoning that it only catches already-known child pornography, and that that person might have a whole cache of novel child pornography. Just look at the drug wars and how many people get arrested for trivial quantities of marijuana. And we hate child pornography way more than drugs.
> As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes.
The judicial system has a presumption of innocence. No one else does, or has any requirement to. The court of public opinion will ruin your life regardless of whether it's true or not.
That would be far less concerning if active cases were sealed, and only published if the defendant was ruled guilty.
> That all said… child porn is really, really terrible and I’m actually in favour of this.
I have a slight disagreement with this. The production of child pornography is obviously abhorrent. This system does nothing to stop that though, because it can only flag already-known child pornography.
This system stops the consumption of already produced and documented child pornography. While it's far outside my own interests, I can see some validity to the argument that consumption is of far less concern because it doesn't directly involve any abuse. The abuse has already happened, and no amount of tracking down pedophiles is going to reverse that. There are second order effects, like potentially reducing the demand for novel child pornography, though. Which leads me to this:
> But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I would imagine this is far more likely to affect consumers than producers. In the drug market, stop and frisk is far more likely to net you users than producers or dealers. For one, there are more users than dealers/producers. For two, dealers, but particularly producers, are typically far more cautious than users.
That's a long-winded way of saying again that this doesn't seem likely to do much in the way of stopping people from abusing children.
But to directly answer your question: doing nothing is absolutely a valid option here. We do that for lots of bad things that happen when the costs of the solution are unbearable. It's what we're currently doing for famine in other parts of the world, which almost surely impacts more children than child pornography could even dream of doing. It's what we do for cars, which again, almost certainly impacts more children than child pornography does.
We wait until we can find a method with benefits commensurate to its costs. A 2016 DoJ report said that since 2002, 10,500 victims have been located and identified by law enforcement. That's 750 per year. There are 74 million children under 18 in the US, so that's 0.001% of children per year. 10,500 people is only 200 people higher than died from "accidental suffocation and strangling in bed" over the same period according to the CDC. The scale of the problem is the same, so should we go ahead and let Amazon send pings to the police if your Alexa thinks your suffocating in bed too?
These are still illegal in the US right now right? It's been a major criticism of these laws that they can hurt people accidentally.
The list ballooned to over a million names, then required an ACLU lawsuit until humans became involved and a review process was initiated.
As long as the majority of Americans dont feel burdened by something, dont expect anything logical to happen.
Which isn't true for Apple. Not too sure about Android. But everything you do on your Apple Devices, including its Hardware, Software and its Services, works more like Apple lend it to you for a price.
And as other mention, only the government and not a private company.
How would that match a hash list of known CP imagery though?
Thought it was buck naked — but that works too. :-)
It's going to be used in China against enemies of the state.
Then it will be used domestically for the same.
Apple is a horrific company that you should stop using.
I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.