It's certainly said to be designed to do it, but have you seen concerns raised in the other thread (https://news.ycombinator.com/item?id=28068741)? There have been reports from some commenters of the NCMEC database containing unobjectionable photos because they were merely found in a context alongside some CSAM.
Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain?
This is overreach.
I wholeheartedly agree: there is an audit question here too. The contents of the database are by far the most dangerous part of this equation, malicious or not, targeted or not. I don't like the privacy implications about this, nor the potential for abuse. I would love to see some kind of way to audit the database, or ensure that it's only used "for good." I just don't know what that system is, and I know that PhotoDNA is already in use on other cloud providers.
Matthew Green's ongoing analysis [0] is really worth keeping an eye on. For example, there's a good question: can you just scan against a different database for different people? These are the right questions given what we have right now.
[0]: https://twitter.com/matthew_d_green/status/14233782854682091...
How many people in the world can certify that it only contains CSAM? I wonder if there are images of US troops committing warcrimes, or politicians doing cocaine, or honeypot intel op blackmail images in there too. Lots of powerful people would love to have an early warning system for leaks of embarrassing non-CSAM images.
Lots of content is shipped in the iOS image but can update independently.
In particular, I’m referencing the figure which says that the database of CSAM hashes is “Blinded and embedded” into the client device. That does not sound like an asset the system remotely updates.
Furthermore, if “an easy to change implementation detail” in your threat model is anything which could be changed by iOS update, you should’ve stopped using iPhone about 14 years ago.
IIRC Wired reported the algorithm "PhotoDNA" worked around 99% of the time a number of years ago, however newer algorithms may be fuzzier. This is not the same algorithm. And even "PhotoDNA" appears to change over time.
I doubt reviewers of such content are at a liberty to discuss what they see or don't with anyone here. Standard confidentiality agreements.
How do we know it hasn't? Maybe the people who have seen these situations have received to national security letters that claim to prevent them from even talking to a lawyer about the non-CSAM classified images they've seen in the course of the investigation?
Maybe pedophiles working for those companies.
Yes, of course, it's "unsubstantiated speculation", but how do you suppose the speculation be substantiated if the entire process and database are not available for audit? That's exactly the problem with this.
I expect people who make positive claims (the database contains innocuous images) to be able to back up those claims. Saying that there are innocuous images in the database is completely different from saying there may be innocuous images in the database.
It would be like if I claimed that there is a burglar in your house - and how do I know? Well all a burglar would have to do is break your window and crawl in.
As with all politically-motivated initiatives that boldly violate the Constitution (consider the FISA Court, and its rubber stamp approval of 100% of the secret warrants put before it), the use and abuse of this system will go largely underground, like FISA, and its utility will slowly degrade due to lack of oversight. In time, even bad matches will log the IDs of both parties in databases that label them as potential sexual predators.
Believe it. That's how modern computer-based gov't intel works. Like most law enforcement policy recommendation systems, Apple's initial match algorithm will never be assessed for accuracy, nor be accountable for being wrong at least 10% of the time. In time it will be replaced by other third party screening software that will be even more poorly written and overseen. That's just what law enforcement does.
I've personally seen people suffer this kind of gov't abuse and neglect as a result of clueless automated law enforcement initiatives after 9-1-1. I don't welcome more, nor the gradual and willful tossing of everyone's basic Constitutional rights that Apple's practice portends.
The damages to personal liberty that are inherent in conducting secret searches without cause or oversight is exactly why the Fourth Amendment requires a warrant before conducting a search. NOW is the time to disabuse your sense of 'daftness'; not years from now, after the Fourth and Fifth Amendments become irreversibly passe. Or should I say, 'daft'?
Apple employs cryptographers, but they are not necessarily acting in your interest. Case in point: their use of private set intersection, to preserve privacy..of law enforcement, not users. Their less technical summary:
> Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child safety organizations. Apple further transforms this database into an unreadable set of hashes that is securely stored on users’ devices.
> Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes. This matching process is powered by a cryptographic technology called private set intersection..
The matching is performed on device, so the user’s privacy isn’t at stake. But, thanks to PSI and the hash preprocessing, the user doesn’t know what law enforcement is looking for.
I understand why this shielding is necessary for the system to work. My point is the crypto is being used to protect law enforcement, not the user.
> I think again, the backlash is daft.
Don't apologize for this bullshit! Don't let your love of brand trump the reality of what's going on here.
Machinery is being put in place to detect what files are on your supposedly secure device. Someone has the reins and promises not to use it for anything other than "protecting the children".
How many election cycles or generations does it take to change to an unfavorable climate where this is now a tool of great asymmetrical power to use against the public?
What happens when the powers that be see that you downloaded labor union materials, documents from Wikileaks, or other files that implicate you as a risk?
Perhaps a content hash on your phone puts you in a flagged bucket where you get pat downs at the airport, increased surveillance, etc.
The only position to take here is a full rebuke of Apple.
edit: Apple apologists are taking a downright scary position now. I suppose the company has taken a full 180 from their 1984 ad centerpiece. But that's okay, right, because Apple is a part of your identity and it's beyond reproach?
edit 2: It's nominally iCloud only (a key feature of the device/ecosystem), but that means having to turn off a lot of settings. One foot in the door...
edit 3: Please don't be complicit in allowing this to happen. Don't apologize or rationalize. This is only a first step. We warned that adtech and monitoring and abuse of open source were coming for years, and we were right. We're telling you - loudly - that this will begin a trend of further erosion of privacy and liberty.
Source (pdf): https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
Or so they say? Did you see the source code?
Anyway that's not the issue.
The issue is what they are going to do next without telling you.
The issue is installation of spyware engine, not it's current usage.
Yet. The point is the new system makes it feasible.
Today was Apple's first press release about it.
Apple certainly didn't want it to be revealed like this, and they might have kept the pieces not related to child privacy hush hush.
TIL - (2014) PhotoDNA Lets Google, FB and Others Hunt Down Child Pornography Without Looking at Your Photos
https://petapixel.com/2014/08/08/photodna-lets-google-facebo...
Fighting to preserve a freedom is not daft, even if it is David vs. Goliath's bigger, meaner brother and his friends.
Would it be ok to use this approach to stop "terrorism"? Are you ok with both Biden and Trump defining that list?
They should just use their own computers to do this stuff.
Also, I pay for iCloud... but they're not paying me for using my phone and bandwidth. I never agreed to that.
They can't just pull the rug out from me after we already have an agreement. I mean they can, because I probably "agreed" to it and some fine print garbage EULA, but those fall apart in a court of law.
You can use another photo app, link it to another cloud provider, and be free of the burden.
If you use Photos, you're along for the ride, and you've consented to whatever it does.
You don't get a line-item veto on code you choose to run, that's never been how it works.
For what it's worth, I'm basically with the EFF on this: it looks like the thin end of a wedge, it sucks and I'm not happy about it.
But being histrionic doesn't help anything.
> it looks like the thin end of a wedge, it sucks and I'm not happy about it.
But you just said that you can disable it. So, you can be upset about a possible future but I'm the one being melodramatic? That's a good one! You chose to run the OS. So by your logic that gives you no right to complain either.
I do understand that I can disable iCloud for my photos and I will do that. We'll see how long that lasts until they decide to tie the next feature and the next one and the next one to something that I don't like. Because that's how this works. Every time they do something that they know people won't like, they simply make it so that you lose access to something else if you decide to stand up for your principles.
This is part of the integrated experience
Apple remains the most privacy respecting major vendor. The only way to do better is fully open software and open hardware.
1. PhotoDNA is already scanning content from Google Photos and a whole host of other service providers.
2. Apple is obviously under pressure to follow suit, but they developed an on-device system, recruited mathematicians to analyze it, and published the results, as well as one in-house proof and one independent proof showing the cryptographic integrity of the system.
3. Nobody, and I mean nobody, is going to successfully convince the general public that a tool designed to stop the spread of CSAM is a "bad thing" unless they can show concrete examples of the abuse.
For one and two: given the two options, would you rather that Apple implement serverside scanning, in the clear, or go with the on-device route? If we assume a law was passed to require serverside scanning (which could very well happen), what would that do to privacy?
For three: It's an extremely common trope to say that people do things to "save the children." Well, that's still true. Arguing against a CSAM scanning tool, which is technically more privacy preserving than alternatives from other cloud providers, is an extremely uphill battle. The biggest claim here is that the detection tool could be abused against people. And that very well may be possible! But the whole existence of NCMEC is predicated on stopping the active and real danger of child sex exploitation. We know with certainty this is a problem. Compared to a certainty of child sex abuse, the hypothetical risk from such a system is practically laughable to most people.
So, I think again, the backlash is daft. It's been about two days of the announcement being public (leaks). The underlying mathematics behind the system has barely been published [0]. It looks like the EFF rushed to make a statement here, and in doing so, it doesn't look like they took the time to analyze the cryptography system, to consider the attacks against it, or to consider possible motivations and outcomes. Maybe they did, and they had advanced access to the material. But it doesn't look like it, and in the court of public opinion, optics are everything.
[0]: https://www.apple.com/child-safety/pdf/Alternative_Security_...