> Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain?

I wholeheartedly agree: there is an audit question here too. The contents of the database are by far the most dangerous part of this equation, malicious or not, targeted or not. I don't like the privacy implications about this, nor the potential for abuse. I would love to see some kind of way to audit the database, or ensure that it's only used "for good." I just don't know what that system is, and I know that PhotoDNA is already in use on other cloud providers.

Matthew Green's ongoing analysis [0] is really worth keeping an eye on. For example, there's a good question: can you just scan against a different database for different people? These are the right questions given what we have right now.

[0]: https://twitter.com/matthew_d_green/status/14233782854682091...

Do the people authorized to query the database have access to view its contents?

How many people in the world can certify that it only contains CSAM? I wonder if there are images of US troops committing warcrimes, or politicians doing cocaine, or honeypot intel op blackmail images in there too. Lots of powerful people would love to have an early warning system for leaks of embarrassing non-CSAM images.

Matt should read the release before live tweeting FUD. The database is shipped in the iOS image, per the overview, so targeting users is not an issue (roughly).

Is the database frozen or can they push out updates independently of iOS updates? If not, targeting individual users definitely doesn't seem possible unless you control OS signing.

The database is shipped in the iOS image.

That's what you wrote originally - and to me it doesn't indicate whether it can also be updated from other sources or not.

Lots of content is shipped in the iOS image but can update independently.

The technical summary provides a lot of detail. I don’t think Apple would omit remote update functionality from it if such capability existed, especially since database poisoning is a real risk to this type of program. I’m comfortable with interpreting the lack of evidence as evidence of absence of such a mechanism. Explicit clarification would certainly help though, but my original point stands: there is positive evidence in the docs which the FUD tweets don’t engage with.

In particular, I’m referencing the figure which says that the database of CSAM hashes is “Blinded and embedded” into the client device. That does not sound like an asset the system remotely updates.

Do you not see any scenerio where the CIA/OGA inserts a hash into the database to specially target one person or a group of people?

I agree database poisoning is a legitimate threat! Including the database in an iOS release (so it can’t be targeted and updated out of band) mitigates it somewhat. At the end of the day, though, more should be done to make NCMEC’s database transparent and trustworthy. And other databases too, if Apple decides to ship country-specific blacklists.

I personally don't believe this process can be made to be trustworthy enough while still serving its stated purpose. It will always remain opaque enough that it could and will be used to violate civil rights.

I don’t see how this system is particularly less trustworthy than the existing system of iOS updates.

You should understand the difference between a protocol and an easy to change implementation detail before throwing around words like FUD.

Has Matt redacted any of the FUD from his tweets last night which aren’t true given the published details from today? For example, his claim that the method is vulnerable to black box attacks from GANs isn’t applicable to the protocol because the attacker can’t access model outputs.

Furthermore, if “an easy to change implementation detail” in your threat model is anything which could be changed by iOS update, you should’ve stopped using iPhone about 14 years ago.

That’s a problem with NCMEC, not Apple’s proposal today. Furthermore, if it were an actual problem, it would’ve already manifested with the numerous current users of PhotoDNA which includes Facebook and Google. I don’t think the database of known CSAM content includes photos that cannot be visually recognized as child abuse.

Why do you not think that? As far as I understand, there is no procedure for reviewing the contents, it is simply a database that law enforcement vouches is full of bad images.

NCMEC, not law enforcement, produces a list of embeddings of known images of child abuse. Facebook and Google run all photos uploaded to their platforms against this list. Those which match are manually reviewed and if confirmed to depict such scenes, are reported to CyberTip. If the list had a ton of false positives, you think they wouldn’t notice that their human reviewers were spending a lot of time looking at pictures of the sky?

It's well-known that this algorithm doesn't have a perfect matching rate. It'd be easy to presume that any false positives are not erroneously tagged images, but the error rate of the underlying algorithm, if all the images were tagged correctly. Who would know?

IIRC Wired reported the algorithm "PhotoDNA" worked around 99% of the time a number of years ago, however newer algorithms may be fuzzier. This is not the same algorithm. And even "PhotoDNA" appears to change over time.

I doubt reviewers of such content are at a liberty to discuss what they see or don't with anyone here. Standard confidentiality agreements.

> it would’ve already manifested

How do we know it hasn't? Maybe the people who have seen these situations have received to national security letters that claim to prevent them from even talking to a lawyer about the non-CSAM classified images they've seen in the course of the investigation?

>> Who audits these databases?

Maybe pedophiles working for those companies.

"Reports from commenters" = unsubstantiated speculation. Weird how no one was able to specifically state any information about these unobjectionable photos except for a theoretical mechanism for them to find their way into the database.

Some have been able to give specific information about these photos: https://news.ycombinator.com/item?id=28071047

Yes, of course, it's "unsubstantiated speculation", but how do you suppose the speculation be substantiated if the entire process and database are not available for audit? That's exactly the problem with this.

That isn't specific information - that is more speculation.

I expect people who make positive claims (the database contains innocuous images) to be able to back up those claims. Saying that there are innocuous images in the database is completely different from saying there may be innocuous images in the database.

It would be like if I claimed that there is a burglar in your house - and how do I know? Well all a burglar would have to do is break your window and crawl in.