A corrupt government has to know they want you "to just get you". Instead they will embed a collision in anti-government meme. That collision will flag you, and now they know you harbor doubts and will come get you.
This is why it's a privacy concern. It's no the tech (like you said photo dna's been about forver), it's the scanning of the phone.
After all, they want to suppress dissent, and they can't catch and arrest everyone - it's much more effective if people don't even dare to voice their dissent.
This is also why you see people in situations like the Arab Spring use known-insecure/monitored means of communication, because they realize that the value and power that comes from communicating and finding other likeminded people is worth painting a target on yourself (because you can't succeed without it, and if you succeed, there will be too many to prosecute).
[1]: https://www.pcmag.com/news/fbi-sold-criminals-fake-encrypted...
Only if that government has worked out some deal with Apple whereby such an anti-government meme would end in the government being notified accordingly. Don't forget that you need a sufficiently high number of collisions, for one, and that those collisions are audited by Apple before being sent to law enforcement.
Photos people send me to my Android are automatically sent through 3rd parties, either through MMS, Facebook messenger, Google Photos, or One Drive. Photos arriving on my device are almost guaranteed to be uploaded to both OneDrive and Google Photos based on how defaults of Android phones are setup.
So someone could already send hash collisions my way (purposely or inadvertently) and authoritarian governments already have access in their respective clouds (at least China does).
And yet, there are not stories of people being falsely accused of child porn due to PhotoDNA hash collisions.
Why does "on device for apple devices only" change the calculus.
You don't even have to implement the feature. All you need is really good proof that you were asked to, and now your job at Apple is secure, along with a huge raise, for years if not for life. Something commensurate with what you and they both know they would lose in the PR damage and possible EU fines.
Also, they'd probably not use a Cupertino developer. I'm sure a dev in a nation with a lot less rights is easier for this sort of work. Find a nation where the protections for employees are worse and good jobs harder to find.
Until the EU makes it a legal requirement. Which they're getting close to
Not without the risk of it being discovered (either through a leak or because someone analyzes the software on the phone), and then having a much bigger scandal on hand.
Everyone suspected they were scanning images for this and a number of other things on their services.
"Following the 2004 Madrid train bombings, fingerprints on a bag containing detonating devices were found by Spanish authorities. The Spanish National Police shared the fingerprints with the FBI through Interpol. Twenty possible matches for one of the fingerprints were found in the FBI database and one of the possible matches was Brandon Mayfield. His prints were in the FBI database as they were taken as part of standard procedure when he joined the military."
"The FBI described the fingerprint match as '100% verified'."
Sure, the probability is lower than getting struck by lightning. I certainly don't play in the rain and I won't be using cloud services where I'm exposed to this kind of nonsense with the FBI.
I don't think anyone can actually know that, because I don't think statistics are kept on how often these sort of dragnet programs result in civil liberty violations and secret grand juries. That should be the more immediate concern, because after that point you are depending on the goodwill of prosecutors... which is a super bad idea.
Just reading those words is rage-inducing, but I'm grateful to have learnt this example of government lying. I feel like it should become an expression that societies teach to their children to warn them about abuses of power. Other mottoes synonymous with government deception and corruption come to mind, but at the risk of being too controversial I will share only their initials and dates: "SAARH" (2013), "MA" (2003), "IANAC" (1973), "NHDAEMZE" (1961), "AMF" (1940).
Nice big clean sample? It probably actually matches. Small sample? Mixed with other people's DNA? Especially in a place they tend to visit (or live in?)?
That, combined with a (pretty reasonable) expectation that the pool of sus hashes will greatly expand as law enforcement and others begin to understand what a convenient side-step this is for due process.
In addition, “pretty reasonable” is an opinion not fact. Where is the evidence that PhotoDNA hashes have been compromised in this way in the fifteen years they’ve been used?
First, how many people really understood this previously? Did society at large actually knowingly accept the current state of things, or did it just happen without most people realizing it? Even here on HN where we'd expect to find people way more knowledgeable about it than in general I'm not sure how well known it was about what was actually happening, though I'd assume most would be aware it was possible.
Secondly, there's a significant difference between your own device or Apple's server doing this. On the technical side of things, right now, it might not matter that much since it currently is limited to things you upload to iCloud. But more philosophically, it's your own device being turned against you to check you for criminal behavior. That's very different from somebody else checking up on you after you willingly interact with them.
This literally only works once you willing send photos to iCloud.
Wait, what?
https://ec.europa.eu/transport/themes/its/road/action_plan/e...
It's like announcing to your family member you're going to tell your neighbor you committed a crime and your family member turns you in first. Yeah, you could expect your neighbor to do the same, but are you really not going to feel any differently about the fact it was your family that turned you in?
Even Pegasus wasn't that much rotten, it at least wasn't indiscriminately installed onto everyone's phone.
But what can I say, there wasn't much uproar about piracy tax on blank CD-R(W) media back in the day, so why not have that now. And eventually we go peak USSR where everyone and their dog is suspect and whoever is arrested is the enemy of the people. Yay, it's somehow reassuring to know I won't live long enough to see it.
I can tell you from working in this area that it's possible for someone to have their lives ruined by a misplaced investigation, have that investigation abandoned because they turn out to be obviously innocent, and for that to not be well-known, because people simply would not understand the context.
Before this Apple scandal, if you'd written to your reepresentative or a journalist or an activist group and said "I was framed for child abuse because of computer program that misidentified innocent pictures", they would attach a very low priority to dealing with you or publicising this. And almost all people who have experienced this kind of nightmare really don't want to re-live it in public for some tiny possibility of real justice being served for them, or for others. They just want it to all go away.
Maybe it's because Google and Microsoft and others' policy of security through obscurity actually succeeded in preventing the details of PhotoDNA from coming to light, and it took Apple exposing their hashing model to reverse engineering by including it on the device for people to finally wake up.
- PhotoDNA
- CSAM scanning on cloud photo platforms
- the acronym "CSAM"
Before this whole Apple client-side scanning debacle... seems pretty likely. A lot of privacy-focused people also avoid Google and Microsoft cloud services like the plague and trusted Apple up to this point to protect their privacy. The fact that Apple was (and is) scanning iCloud Photos libraries for CSAM unbeknownst to most of us is just another violation of that trust and shows just how far the "what happens on your iphone, stays on your iphone" privacy marketing extends (read: not past your iphone, and sometimes not even on your iphone).
"U.S. law requires tech companies to flag cases of child sexual abuse to the authorities. Apple has historically flagged fewer cases than other companies. Last year, for instance, Apple reported 265 cases to the National Center for Missing & Exploited Children, while Facebook reported 20.3 million, according to the center’s statistics. That enormous gap is due in part to Apple’s decision not to scan for such material, citing the privacy of its users." [1]
[1] https://www.nytimes.com/2021/08/05/technology/apple-iphones-...
Is that really true? My understanding is that the manual reviewers at Apple only see some kind of low-resolution proxy, not the full-resolution image. I'd also be shocked if the human reviewers were shown the original, actually CP image, to compare to.
Given that, it's not necessary to produce an actual visual match, it's just necessary to produce an image that when scaled-down looks like CSAM (e.g. take an actual photo of a kid at the beach and photoshop in some skin-coloured swimwear with creases in the right places).
> Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America?
The attack I'd worry about here is similar to swatting. Someone who doesn't like me sends a bunch of images like the ones I described above (not just one), they end up synced to iCloud (because Apple wants _everything_ synced to iCloud) and Apple reports me to the authorities, who end up knocking at my door and arresting me.
Even though I'm innocent, I'll probably have most of my computers confiscated for a while and spend a few days locked up.
> PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of.
PhotoDNA's algorithms and hashes aren't public, so it's not clear how an attacker would exploit PhotoDNA in the way that people are afraid will be done for Apple.
PhotoDNA also isn't, as far as I know, part of a product that aims to create unprotected backups of the phones of nearly a billion users. Apple really wants you to upload your whole phone to iCloud. The only comparable alternative is Google's Android backup but Google does the right thing and end-to-end encrypts that.
But a bigger concern is automatically. WhatsApp for example can automatically save all received photos to your Camera Roll, which is of course automatically backed up to iCloud for many people. So an attacker could potentially just send you a bundle of 40 images of however many and your phone automatically sends it to Apple.
I’m guessing most times this happens, the accused try to keep it on the DL.
https://www.wmur.com/article/fbi-raids-radio-talk-show-host-...
I understand that the system as stated today has multiple safeguards against such things happening but...
Given the sum-total of bs I've seen happening in this country, yes, believe that kind of thing is quite possible.
Not necessarily. The FBI knocks on your door when they have convinced a warrant-signing judge, that’s there is probable-cause, or that additional information can be collected to build a case in which the defendant will ‘plead’ or make a deal for sentence reduction. 90% of defendants make a deal and never go to trial because by that time the stakes are so high a guilty verdict includes the harshest possible sentence.
FBI only needs to convince you they can win a case or the judge that there might be fruit behind a locked door, they don’t need direct evidence. It’s really up to the judge.
And we are not talking about some underpaid contractors making life altering decisions.
And we are also not talking about a system where just a false accusation can and will destroy one's life irreparably.
Specificly the person doing the scanning already has access to photos and can double check the results without having to sieze the device, a rather public process.
being on a government maintained secret list of CSAM flagged would probably have a lot of consequences in your life going forward without you knowing about that. It may be just one innocent image which accidentally matched the hash, yet without any procedure to get you off that list or even just to learn whether you're on that list ...
>If some corrupt government wants to get you they don’t need this.
it isn't corrupt government which is a threat here. It is a well intended well functioning relatively uncorrupt government which spends a lot of effort to protect its society from terrorism, child porn, sex trafficking, drugs, etc. In case of corrupt government the situation is kind of easier - you can always learn and correct necessary things through the corrupt government officials by paying a corresponding "fee".
I believe a single image flag could be used as pretext to arrest someone that the government has already deemed an enemy of the state. Ex. Someone like Julian Assange
https://en.wikipedia.org/wiki/Paul_Reubens#2002_pornography_...
-> if some corrupt government wants to *FIND YOU SHARING IDEAS THEY DON'T LIKE* they can *DEFINITELY* use this
- You receive some naughty (legal!) images of a naked young adult while flirting online and save them to your camera roll.
- These images have been made to collide [1] with "well known" CSAM images obtained from the dark underbelly of the internet, on the assumption that their hashes will be contained in the encrypted database.
- Apple's manual review kicks in because you have enough such images to trigger the threshold.
- The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side.
- Your case is forwarded to the FBI, who now have cause to turn your life upside down.
This scenario seems entirely plausible to me, given the published information about the system and the ability to generate collisions that look like an arbitrary input image, which is clearly possible as demonstrated in the linked thread. The fact that most of us are unlikely to be targets of this kind of attack is little comfort to those that may be.
[1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
Given how most companies have barely paid, clueless and PTSD suffering human reviewers, a litany of mistakes is to be expected.
We should expect all the coherence of the twitter's nipple policy, except now it put you in jail or at least ruins your life with legal fees
The law states that if you find an image that's believed to be CSAM, you must report it to the authorities. If Apple's model detects CSAM on the device, sending the whole image to the moderation system for false positives carries the risk of breaking the law, because the images are likely going to be known CSAM, since that's what the database is intended to detect, so they'd be accused of storing it knowingly. Perhaps that's why the thumbnail system is needed.
So why wouldn't Apple store the files unencrypted and scan them when they arrive? That would mean Apple would remove themselves from liability by preventing themselves from gaining knowledge of which images are CSAM or not until they're scanned for, but could still send the original copy of the image with a far lower chance of false positives when something is found. That knowledge or the lack of it about the nature of the image is the crucial factor, and once they believe an image is CSAM they cannot ignore it or stop believing it's CSAM later.
That question may hold the answer to why Apple attempted to innovate in how it scans for child abuse material, perhaps to a fault.
Not to mention your scenario requires someone you trust trying to "get you." If that's true, then none of the other steps are necessary since you're already compromised.
And as far as your point about "someone you trust trying to 'get you'"... have you ever dated? Ever exchanged naughty photos with somebody (I expect this is even more popular these days among 20-somethings since covid prevented a lot of in-person hookups)? This doesn't seem crazy for a variant of catfishing. I could easily see 4chan posters doing this for fun.
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
Threshold Secret Sharing Synthetic Match Vouchers
Not they haven't. The files are still uploaded to Apple's servers where Apple holds their encryption keys. Apple or the FBI could scan the photos for CSAM whenever they want to.
All because you clicked on a link.
Phishing attacks can now put you in prison, label you as a pedophile and sex offender, and destroy your life.
a) That you know of b) Apple won't be doing this either
> Apple is building it.
No, they aren't.
PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of.
If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’ve done something bad without evidence and imprison you. It happens all the time. It’s even happened in America. Just look up DNA exonerations - people have had DNA on the scene that literally proves their innocence and they’re still locked up.
People should care about their governments being corrupt, not necessarily about this.