Don't feel bad at all. I dumped macOS entirely from production workflow. I cannot work on computer knowing that something is "scanning" me and I am glad that my "paranoid" feeling stopped me to upgrade all office macs.

Billionaires at (Apple) don't give a flying f*ck about users privacy. It is all vertical integration in the name of world domination. How removed from reality they are. This is week after Pegasus/NSO and there is no law who requires them to "scan" on device. So the logical explanation is "growth". Way to go Apple.

>> there is no law who requires them to "scan" on device.

There is. Apple must comply with warrant requests. If they have a system for scanning files on customer devices they must, if presented with a warrant, allow police access to that system. We can quibble about jurisdictions and constitutional protections, but if the FBI shows up with a federal warrant demanding that Apple remotely scan Sandworm101's phone for a particular image, Sandworm101's phone is going to be scanned.

That's an incomplete statement.

Currently, they must comply with warranty requests by scanning if they have the ability to scan.

If they have no such ability (say, because they designed their phones from a privacy-first perspective), the law makes no requirement that they create such a capability.

And that's what pisses people off about this.

If Apple launches a system for comparing iCloud uploads to a third-party hash list, then adding the ability to do targeted scans for arbitrary additional law-enforcement-provided hashes would also be a form of creating capability.

The people getting pissed off about this have not, so far as I’ve seen, demonstrated why the law would require Apple to add the capability for targeted scans of arbitrary hashes.

Police can use a warrant to ask you for a video file they suspect you have. They can’t use a warrant to force you to videotape someone—even if you already own a video camera. The same principle applies to Apple.

>> They can’t use a warrant to force you to videotape someone

Maybe in the narrow context of US domestic child pornography investigations. In the wider world it is very possible for police to force such things. Even in the US, CALEA demands that certain companies develop abilities that they would not normally want (interception). The principal that US companies need not actively participate in police investigations disappeared decades ago. On the international level, all bets are off.

https://en.wikipedia.org/wiki/Communications_Assistance_for_...

"by requiring that telecommunications carriers and manufacturers of telecommunications equipment modify and design their equipment, facilities, and services to ensure that they have built-in capabilities for targeted surveillance [...] USA telecommunications providers must install new hardware or software, as well as modify old equipment, so that it doesn't interfere with the ability of a law enforcement agency (LEA) to perform real-time surveillance of any telephone or Internet traffic."

Can you explain why you think CALEA would apply to what Apple announced?

And why such application would have had to wait until Apple announced this? In other words, if the law can force Apple to do things in general, why does the law need to wait for Apple to announce certain capabilities first?

>> why you think CALEA would apply to what Apple announced?

Read what I stated. I said no such thing. I said that CALEA shows that companies can sometimes be forced to be do things they do not want to do, to take actions at the behest of law enforcement that they would not normally do. CALEA would clearly not be applied to apple in this case. It would be some other law/warrant/NSL that would force apple to do something it normally would not do. CALEA stands as proof that such things have long been acceptable under US law.

I understand why you are worried that a more powerful law could be passed by Congress in the future.

I thought this particular conversation was about what Apple can be forced to do by a warrant issued under current law.

> why the law would require Apple to add the capability for ...

This isn't the sort of thing that is mandated by law, but rather requested behind the scenes by espionage, er, "law enforcement" agencies. They might be more or less friendly deals; nice monopoly you have here, it would be a shame if something happened to it...

Can a warrant compel them to develop the capability?

A warrant alone, very likely not. A court order from a judge (slightly different than a warrant), possibly not nut it's not exactly a well tread area of legislation/law. It would probably run afoul of the first amendment given some readings, but it would have to be tested in court and would likely be argued that requiring a backdoor via some law doesn't cause compelled speech or that making that backdoor isn't protected speech etc. It's a relatively new area to be explored.

Yes. Lavabit. Those warrants demanded that Lavabit alter its system to capture passwords and/or decrypt stored email. Lavbit instead decided to stop operating and delete everything rather than comply. Such warrants have not been fully tested in courts but they do exist.

IIRC they gave up the information AND they shut down

They handed over a hard copy of their private key, a printout. The FBI went to court to demand a machine-readable copy. Then they shut down.

Thanks for that! Didn't know.

Have they been served a warrant?

Just because someone could force you to do something you don't want to do is not really a major concern if you choose, preemptively, to do the thing. The concerning part here is Apple signalling that they have executives that don't see phone scanning as a problem. That is a major black eye for their branding.

Apple's warrant canary disappeared in 2014.

The law gets debatable here. The warrant can ultimately be served only to the owner/responsible party for the system. If apple decides that they own all iPhones, a claim they've loosely made for a long time re Flash etc. Then they could be served a warrant on the device.

If they just sold the device to someone, then the police would have to issue a warrant to that individual. My understanding is that as an individual your options are either to comply with the warrant or commit a separate crime destroying the evidence or refusing the warrant.

Apple does not claim they own all iPhones.

The legal principle is the same, actually. The law says you can install whatever software you want on an iPhone and Apple cannot stop you (jailbreaking is legal). But the law cannot force Apple to make it easy for you to do so.

I haven't kept us with the latest cases, but I'm not sure where it falls on app-based SaaS.

I.e. If I use an app, on a system in which app updates are possible, which encrypts my data locally and stores it on their cloud, then the owner/responsible party for the app is...?

> "the law makes no requirement that they create such a capability."

and they have not created such a capability. This is built into the image upload libary of iCloud. This has less ability to "scan the device" than the iOS updater which can run arbitrary code supplied by Apple and read/write anywhere on the system. This is less able to be extended to a general purpose scanner than that is.

Warrants are only part of the issue. The bigger issue is civil and criminal liability. If someone uploads child porn to iCloud and then shares it with someone else Apple is possessing and distributing child porn.

Today they aren't liable because there is a law that shields them. But that law comes with strings attached around assisting law enforcement. By doing an end run around those strings Apple is not holding up it's end of the bargain and runs the risk of lawmakers removing the liability shield.

If that happens then it's a whole new ball game.

> If someone uploads child porn to iCloud and then shares it with someone else Apple is possessing and distributing child porn.

If someone sends CSAM using Federal Express, is FedEx legally liable for "possessing and distributing" that material? Does FedEx need to start opening up packages and scanning hard drives, DVDs, USB sticks, etc. to ensure that they don't contain any CSAM or other illegal data?

I really struggle with the lengths that people are going to to justify these moves. If we can justify this, it's pretty simple to justify a lot more surveillance as well. CSAM is not the only scourge in our society.

Reminder: Apple tells us that they consider privacy a "fundamental human right"[1]. That simply does not square with their recent announcement of on-device scanning, and some would argue that it does not square with scanning or content analysis anywhere, especially on behalf of government.

[1] https://apple.com/privacy

>If someone sends CSAM using Federal Express, is FedEx legally liable for "possessing and distributing" that material?

Yes:

https://www.dea.gov/press-releases/2014/07/18/fedex-indicted...

That was for drugs but conceptually the same for CSAM.

This is not at all conceptually the same.

FedEx was not held liable simply because their service was used to mail illegal drugs. They were held liable because not only did they know about the specific instances in which it was mailed, they allegedly conspired with the shippers to facilitate the mailings. They were knowingly mailing packages to parking lots where drug dealers would wait to pick them up:

> According to the indictment, as early as 2004, FedEx knew that it was delivering drugs to dealers and addicts. FedEx’s couriers in Kentucky, Tennessee, and Virginia expressed safety concerns that were circulated to FedEx Senior management, including that FedEx trucks were stopped on the road by online pharmacy customers demanding packages of pills; that the delivery address was a parking lot, school, or vacant home where several car loads of people were waiting for the FedEx driver to arrive with their drugs; that customers were jumping on the FedEx trucks and demanding online pharmacy packages; and that FedEx drivers were threatened if they insisted on delivering packages to the addresses instead of giving the packages to customers who demanded them.

They had drug addicts literally stopping FedEx trucks on the street to intercept packages from online pharmacies. I don't see how this specific situation is analogous to Apple at all.

The fact remains that FedEx is not scanning hard drives, DVD or other storage devices going through their network and they don't appear to be breaking any laws by not doing that scanning.

>They were held liable because not only did they know about the specific instances in which it was mailed, they allegedly conspired with the shippers to facilitate the mailings.

Apple knows that CSAM is being sent and conspires to do so (i.e. transmits the image). Conceptually they are the same.

The rest of your post details the practical differences between sending physical packages and digital images.

I'm not even sure where to start here.

FedEx knows that CSAM is sent using its services in the same way that Apple does. That is to say that both companies know that CSAM has been sent historically and both know that is possible to send that type of material using its services, and one could argue that they should assume that their services are used to transfer that data.

Why does one of these companies have to go to such great depths to search out and report these materials and the other does not? If we suggest that Apple "knows" about CSAM on their network, we must also accept that Apple "knows" about many other crimes in which the devices they are sell are used in the planning and execution of those crimes. Why are they only focused on CSAM if they'd also have legal exposure in these other crimes simply for being a hardware and service provider?

Lastly, to suggest that Apple "conspires" to transmit this content is wholly inaccurate. Conspiracy implies two or more parties are in explicit agreement to commit a crime.

>If we suggest that Apple "knows" about CSAM on their network, we must also accept that Apple "knows" about many other crimes in which the devices they are sell are used in the planning and execution of those crimes. Why are they only focused on CSAM if they'd also have legal exposure in these other crimes simply for being a hardware and service provider?

It's not knowledge of a crime. Apple is committing a crime by possessing and distributing CSAM.

>Lastly, to suggest that Apple "conspires" to transmit this content is wholly inaccurate. Conspiracy implies two or more parties are in explicit agreement to commit a crime.

But that is exactly what they're doing. They are agreeing to possess and deliver images of which they know a portion are CSAM.

> Apple is committing a crime by possessing and distributing CSAM.

Apple's liability for content is limited by Section 230. If they find out about specific instances of CSAM on their servers they are required to remove it.

They are not required to hunt for it or otherwise search it out. They are not required to scan for it. Anything currently being done in that regard is voluntary.

> But that is exactly what they're doing. They are agreeing to possess and deliver images of which they know a portion are CSAM.

When you use Apple's cloud services, you agree to multiple legally binding agreements that include provisions about not using their cloud services to store or transmit illegal materials including CSAM.

Someone using Apple's services to do that in contravention of those agreements does not constitute them "agreeing to possess and deliver" that content.

What WOULD constitute them conspiracy and them agreeing to possess and deliver that content would be if they were informed of the specific instance of content being transmitted and delivered and did nothing about it or otherwise enabled its continued storage and distribution.

At least in the United States, this is absolutely false. A warrant cannot force them to do something they have no capability to do.

(a) They created the capability, that's the problem

(b) The world is not the United States

I don’t think this is correct.

1) Even if Apple does roll out this system, it’s not a system for scanning files on customer devices. It’s a system for comparing photos being uploaded to iCloud to a single standardized (everyone has the same one) list of hashes.

2) Apple likely has the legal power to refuse to alter that list of hashes, by the same argument they used against the FBI’s request to bypass the unlock code on specific iPhones.

3) Any argument about what Apple will and will not need to do with this system needs to explain how Microsoft Defender (or other AV products) interact with law enforcement, since those are software systems that scan client devices (ALL files, typically) for signature matches.

> and there is no law who requires them to "scan" on device

Yet. The demands by "concerned parents" (aka fronts for secret services, puritans/other religious fundamentalists and law-and-order hardliners) to "do something against child porn" have grown ever more strong and insane over the last years. (And you can bet that what is used on CSAM will immediately be used to go after legal pornography, sex work, drug enforcement, ...)

Many current and powerful politicians don't understand a single bit about computers, to the point of bragging of never having used one (e.g. Japan's cybersecurity minister) or having assistants print out emails daily and transcribing handwritten responses (can't say more than that this is the situation for at least two German MPs) - but there are young politicians who do, and will replace them over the next decade hopefully. That means it's last chance for lobbying groups to get devastating laws passed through, and we must all be vigorous in spotting and preventing at least the worst of them.

And what is suspiciously lacking in Apple's response: what are they going to do when they are compelled to extend the CSAM scanner by law in the US, India, China and/or EU? It's feasible for Apple to say they'll just be sticking the middle finger towards markets such as Russia, Saudi-Arabia or similar tiny dictatorships, but the big markets? Apple can't ignore these, and especially India and China are heavyweights.

Why would they extend the CSAM scanner? It would be much simpler to just use all the OCR and image classification functions they have already deployed.

CSAM scanning is only useful for areas where Apple really doesn't want to even look at the actual material until they are extremely certain that it's a match. If they want to detect anti-government propaganda or something, there would be no such concerns, they would just do a regex search of your inbox. Infinitely more convenient.

>> useful for areas where Apple really doesn't want to even look at the actual material

Correct. It has plausible deniability built in. Apple is unable to verify that the images the government are looking for are actually CSAM. They could be political. They could be protest images. They could be Winnie the Pooh. Apple can plead ignorance as it blindly scans for whatever the requesting government asks it to scan for.

Nobody really minds that this system is going to be used for CSAM. What everyone recognizes is how ripe this system is for abuse, how easily it can be leveraged by oppressive governments. And Apple can play the innocent.

> Nobody really minds that this system is going to be used for CSAM.

I beg to differ. It doesn't matter how evil the content is, no scanning of my computers by outside parties, period. More so by scanning law enforcement can even plant legitimate child pornography on people's computers and get convictions all the easier because the system self-reports.

It's clearly not the scanning you are opposed to, it's the reporting. Apple apps, well a lot of apps, already scan all your content, for classifying images, doing OCR, extracting dates and addresses etc. That is nothing new, it's omnipresent.

What about the human reviewers at Apple? Those need to confirm matches, and they obviously look at the photos to do so. Apple can’t claim ignorance.

Also, it has to be two requesting governments.

The current Apple policy to use human reviewers. That might change at a moment's notice and there is no technical reason why Apple couldn't bypass humans for certain requests. One must always judge a new system three ways: how it was meant to be implemented, how it is actually implemented, and how it might be abused by bad actors in the future.

Except that Apple will review the photos once you've matched 30 of them, so it's still not possible for the government to misuse it.

So some underpaid contractor reviewing "visual derivatives" that may or may not be CSAM completely prevents governments from misusing this in your mind?

Considering how high profile and incredibly sensitive this is, I doubt they will hire an underpaid contractor. It's the opposite of app review, where the volume is very high but the stakes are low.

Facebook hires underpaid and poorly treated contractors to moderate exactly this same kind of content. I see no reason to believe Apple will be any different, particularly a few months from now when the general public's attention has shifted to other matters (assuming they're even paying attention right now. I don't think the interests of HN are necessarily representative of the general public..)

In China, iCloud is run by the government.

It's actually not, but even if it were, that would be yet another reason CSAM scanning is completely irrelevant for government spying.

Any spying really. It's much easier to just look at the images themselves.

> It's actually not

It's Joe Wong's joke, that it's like peeing in the snow in a dark winter night, while there is a difference but it's really hard to tell.

Why even keep Apple in the loop? Why not just allow government to submit scanning models directly?

Which Apple will dutifully install and run, because they're required by local laws.

> "Which Apple will dutifully install and run, because they're required by local laws."

Which Apple have stated that they won't do, and have designed the system so they can't do that without it being found out: https://news.ycombinator.com/item?id=28221082

Can't you at least post accurate information about this system and support outrage based on facts instead of fantasy?

> Which Apple have stated that they won't do

For your random off-of-the-mill dictatorship, yes.

For the US? EU? China? India? No way they can refuse such a request from these markets. And if they could and get away with it, it would be a very worrying situation in itself regarding (democratic) control of government over global mega corporations.

If you believe "the government can compel any company to do anything", then this system is no worse than any other.

Although how do you think Linus Torvalds "manged to get away with refusing" adding backdoors? https://www.techdirt.com/articles/20130919/07485524578/linus...

Simple reason: every line of code in the Linux kernel passes many eyeballs in the open. Even if someone were to compel Linus Torvalds to commit a backdoor and he would comply with that order, people across many jurisdictions would notice this immediately, rendering the backdoor worthless.

> "and what is suspiciously lacking in Apple's response: what are they going to do when they are compelled to extend the CSAM scanner by law in the US, India, China and/or EU?"

from https://daringfireball.net/linked/2021/08/09/apple-csam-faq - "we will not accede to any government’s request to expand it."

From https://www.msn.com/en-us/news/technology/craig-federighi-sa... - "“We ship the same software in China with the same database we ship in America, as we ship in Europe. If someone were to come to Apple [with a request to scan for data beyond CSAM], Apple would say no. But let’s say you aren’t confident. You don’t want to just rely on Apple saying no. You want to be sure that Apple couldn’t get away with it if we said yes,” he told the Journal. “There are multiple levels of auditability, and so we’re making sure that you don’t have to trust any one entity, or even any one country, as far as what images are part of this process.”" - Craig Federighi

A “government’s request” is different from a country’s law. “This process” is not the only process by which third parties can breach your privacy.

How likely is it that you will have enough colliding images in your photo library to even trigger a review?

I'm guessing you need at least 5 images, perhaps much more, to trigger it. In any case, 1 image is definitely not enough.

For a normal random person, a grey man. In the real world, a single collision could be enough for the police to acquire further access to people they are already looking at. If the police want access to your phone for other reasons (drugs, taxes, illegal speech) they can use that one collision to get a warrant which will give them greater access.

It is akin to cops wanting to search a car. They don't need a warrant. They only need to follow the car until it breaks any number of traffic laws. Then the resulting traffic stops lets them talk to the driver, who seems evasive, which gives them probable cause, which gets them a warrant to perform a full search. The accidental collision, week evidence of an offending image on the phone, opens the door to whatever other investigation they want to do.

Isn't Apple seeding their database with fake hits so that a court can't order them to turn over everyone who's just a few shy of Apple's threshold? So when this database leaks, won't there be a lot of people accused of horrific crimes simply because Apple seeded their database with random hits?

Why would people pay a company to falsely accuse them of horrific crimes?

Since, as I have read, Microsoft and Google already do this, where are all the illicit cop take-downs? I have not heard of any.

They are scanning images on online accounts, stuff that is stored on their systems and system files on machines (files they own). They are not, from what I have read, scanning customer-owned images stored on customer-owned devices.

Every AV (antivirus) software system scans customer-owned files on customer-owned devices. So the question is the same: if it’s easy for law enforcement to deputize such a system, where is the flood of cases built off of evidence gathered this way?

I feel like you are both chasing red herrings.

This is not about child porn nor about what AV software can or cannot do.

It's about normalising mass surveillance and implementing populace control. They don't want another Snowden to scare the public with reports about "backdoors" and mass privacy violations.

They want the coming generation to perceive it as normal. Because all phones do it, hasn't it always been this way, and think of the children.

Oh, these dissidents in $distant_country that will be muted and killed using Apple's shiny surveillance tech? Well, evil governments do what they do. But we are not like that. Over here it is only about the child predators. Trust us.

Apple has been an opponent of these developments for decades. Now they are spearheading it.

>> Every AV (antivirus) software system

Mine doesn't. If Ubuntu or ClamAV is scanning all my photos and reporting the results to Canonical against my will then I will soon be having words with Mr. Linus.

If law enforcement can force software companies to adapt their software to serve law enforcement purposes, why aren’t Ubuntu or ClamAV doing so already? What makes them better at resisting requests from law enforcement than Apple?

>> why aren’t Ubuntu or ClamAV doing so already?

Because if some local police agency like the FBI were altering code on an open source project as large as ubuntu then the world would know about it pretty quickly. The linux community would burn every bridge with Canonical and ubuntu would disappear overnight. The idea of the FBI adding malware to ubuntu unnoticed, malware that openly scans files and reports to remote servers, is the stuff of comic books.

Canonical and ClamAV aren't trying to curry favor with the US government to stave off antitrust action.

Apples solution only scans photos that will be uploaded, and the results of the scan require a server side to be evaluated. The client does not know whether a match occurred.

I'm not sure what you mean. If we are talking about Apple's new feature, they will not even be alerted until you have enough matches.

And the police probably won't be using any neural hashes if they have access to your device, so I'm not sure what you are talking about.

That is an Apple current policy, not any sort of law. If the FBI wants to know if there are any collisions on a particular phone, that will be shared. If the FBI wants to know of all single collisions, that too must be shared. A corporation's internal policy decisions are nothing when faced with a government official carrying a warrant. If they have data indicating possible crimes, no matter how small, they can be forced to divulge it to investigators.

Unlike Lavabit, I don't think Apple will ever close up shop to protect customer data.

Well of course if you believe that FBI can ask Apple to share anything and they will agree, and not tell us, then what is even the point of this discussion? Then they already have access to every photo, every email, every text. Because Apple controls those apps on your phone.

The difference is that now they confirmed that they do search your private images.

Why would that make it any likelier that they are also secretly searching your files, with another method? Seems completely orthogonal to me.

They showed that they are willing to do it in one case. Why not in other cases?

I thought the system is designed such that the threshold must be met to check the matches, i.e. it’s not possible to follow the FBIs request without changing the entire system.

int matchThreshold = 30;

if (numMatches > matchThreshold) { snitch(customer) && customer.life.ruin() }

For individual users the risk is very small, but when you have a billion users the chances of innocent people being caught up is pretty much 100%.

Platform owners like Google, Apple, Twitter and Facebook should really keep stuff like that in mind when they deploy algorithmic solutions like this.

Like dhosek said, the manual review step should reduce the risk quite a bit, though.

They did keep it in mind, that's why they require 30 matches, which gets the false positive rate down to 1 in a trillion per year, so on average it will happen about 1 per millennium, given a billion users.

So that's per photo library, not per photo. The rate per photo in their testing was 3 in 100 million, then they added a 30x safety margin and assumed it's actually 1 in a million.

https://www.zdnet.com/article/apple-to-tune-csam-system-to-k...

Not sure why the downvotes. You are absolutely correct that a threshold in the number of positives (false or otherwise) must be met.

To be fair though we do not know what the threshold is. But I would guess even higher than 5 — I would presume 12 or more.

I'm no criminologist (IANAC) but when you read about someone getting busted with child pornography they have hundreds or thousand of images — not one, not five. They're "trading cards" for these creeps.

Apple says they expect to use 30 as the threshold.

Apple has publicly stated 30. Plus, there would be a manual review of the images before forwarding to law enforcement. I don't see this collision attack leading to innocent people being handed over to the police. The worst-case (best-case?) scenario is that Apple gives up on the whole hashing system and then there will be zero chance of there ever being end-to-end encryption of photos in iCloud.

I saw 30 mentioned somewhere, and I think that’s a more likely guess than 5. When men have a porn collection, it rarely has less than a hundred images, and so the threshold for CSAM detections can be set quite high in quantity, and enjoy a near-zero false positive rate aside from malicious hackers.

(And yes, 90% of CSAM abusers are men, so y’all will have to find some other way to weaken my argument.)

I actually want Apple to stand ground and implement this feature. Like you said the double down on PR and marketing was enough for me. I may not be dumping all iOS and Mac for now. But it was "the" definite signal and evidence this is no longer the old Steve Jobs's Apple. It is like watching Mark Zuckerberg talking about privacy when he doesn't understand anything about it. ( Or more like he has a different understand of privacy than most people )

If they stand ground it makes the decision a lot easier for others. If they dont they will continue to use the boiling frog tactics.

After all they dont think they are wrong and they are still righteous. And It would be far more interesting to see how the world folds. The whole computing market, Google, Apple and Microsoft are currently operating with a moat that is literally impenetrable. This will be a small push towards a new world of possibilities.

Yes. I really hope they stand ground. I wish I won the lottery I could put a few million into the Social Media echo chamber to support this. Or you know, cough certain Apple competitor could do this.

Chances are by the next sales report it will be forgotten. Let's see how deep the memory hole goes.

"Steve Jobs Apple"? I don't think Jobs would give a damn about people crying about Apple's decisions. I don't know why people thinks he would be a smidgen better than whoever managing apple after him.

Because he actually understand privacy better than 99.9% of people in Silicon Valley. He is also a product person who understand how users feel. Compare to current Apple which is "still" trying to give me a technical explanation of what is and what's not.

Steve Jobs would have implemented this in secret and never told us at all, same as they already did for iCloud photos so many years ago. That would have been a far better approach than today’s Apple is taking. Oh well.

There’s a significant leap from implementing something server side to on the consumer handheld devices themselves. Even if it’s just similar software it is regardless much more serious.

I personally thought the unencrypted backups was enough of a death-knell as it provides everything on your phone but anything with real-time on device access is always a gold mine for surveillance hawks.

It's serious to you, but CSAM scanning is irrelevant to 99.99999% of Apple's customers, and Jobs would never have allowed an announcement about migrating CSAM scanning from uploads to the cloud to the device uploading to the cloud. That's an implementation detail that wouldn't be relevant to discuss with outsiders. Instead, I expect he would have presented it in a closed session to the FBI and/or Congress. Never to the press, not like this.

> CSAM scanning is irrelevant to 99.99999% of Apple's customers

How long until an group of governments tells Apple to add Tank Man to the list?

Why would they bother? That's a terrible way to approach it.

Just pass legislation requiring in-country datacenters that can be decrypted by thoughtcrime enforcers, like Russia and China are doing. Trying to get this done via a CSAM list that's absurdly closely audited would be a huge waste of time and not provide any significant benefit, and if such a request were ever made public, would likely result in severe political and economic sanctions.

That's what everyone's missing in this argument. There's no need to be all underhanded and secretive when you can just pass laws and conduct military-backed demands upon companies using those laws. Trying to exploit the CSAM process would be a horrifically bad idea, and would result in public exposure and humiliation, rather than the much more useful outcome that simply passing a law would provide.

Without the technology deployed, Apple can (and did) say they don't have the ability to break into users' phones.

If Apple deploys on-phone scanning, governments can just tell Apple to support a new list. It won't be the NCMEC CSAM list. It will be a "public safety and security" list. I wouldn't rule out underhandedness either. [1]

[1] https://www.nytimes.com/2020/07/01/technology/china-uighurs-...

Apple already has technology deployed to perform binary file scans of every file on macOS and iOS, and the ability to at any time release signatures for those scans, that are very difficult for normal users to prevent updates for. They've had that for years, maybe even a decade by now, and so far to date we have seen no abuse of that list.

How is Apple's new CSAM list somehow increasing the chances of Apple going rogue, given that we've all been living with that risk for the past X years?

What technology are you referring to as already deployed?

For macOS, I'm talking about XProtect and MRT. I don't know the exact subsystem names on iOS, apologies.

https://support.apple.com/guide/security/protecting-against-...

Each system is closed source, provides a mechanism for checking content signatures against files on disk, and is thought to report telemetry to Apple when signatures are found.

How is CSAM scanning new and different from those existing closed-source systems?

I'd say the primary differences are that the CSAM scan is a perceptual hash rather than a regular file hash, and that the technical infrastructure of the CSAM system is designed from the ground up to be used against (rather than for) the user and report them individually to authorities for violation.

Do you have an alternate design in mind that is both "used for the user", and is also effective at reporting CSAM content being uploaded from the device, without allowing CSAM abusers to opt-out of that reporting? I haven't been able to come up with anything myself, but maybe you've had better luck.

I can only point to other people who know more than me.

https://stratechery.com/2021/apples-mistake/ is a smart tech commentator

https://www.nytimes.com/2021/08/11/opinion/apple-iphones-pri... are two security/encryption experts

Exactly this. I see lots of people saying that Apple are forced to implement something via legislation as if it’s an excuse for it.

If they want to do business in China they will be forced by their legislation.

My entire argument is don’t build the mechanism.

China forced Apple by legislation to implement new iCloud algorithms for assigning China-region user data into China-hosted datacenters. Most countries, unlike the US, are not constrained by a requirement to only exercise previously-built mechanisms and not create new ones, in response to government demands. If China decides to require Apple to censor non-CSAM content on-device, they will do so whether or not CSAM content fingerprinting exists. That China has not done so is because they benefit greatly from Apple's manufacturing and sales and do not wish to create a diplomatic incident with Apple.

> China-hosted China-decryptable datacenters.

China hosted, yes, but Apple denies China-decryptable, so that’s speculation unless you have a good source.

Nope, that's just me remembering wrong. Deleted those two words, thanks for the correction :)

until ios source code is closed all privacy claims is only backed by trust. They easily can do whatever they want if you're not compiling from source. There's no way to ensure your data is not leaving your iphone/mac with some "system" network requests.

Security researchers and hackers routinely look at the code on the device, though.

Well Steve Jobs was the one that resisted PRISM and gave the middle finger to NSA.

NSA PRISM was and illegal and warrantless text analysis and search system for most communications on the Internet, collected all communications without bothering to filter at all, and had no protections in place to prevent people from randomly searching and reading content out of human curiosity.

Apple's CSAM implementation protects the user against algorithm defects, does not expose the user to legal trouble until they have at least 30 human-verified visual matches of CSAM content, and occurs on-device using a small set of confirmed and audited signatures to ensure that CSAM scanning requires a central system only for verifying the unverified, blurred, positive matches.

I would hesitate to compare Steve Jobs' views on PRISM to his likely views on something that is so clearly opposed to it in so many ways. So I do not yet understand your viewpoint that Apple CSAM scanning and PRISM would have been treated equally by Steve. Help me understand?

It's not really end-game, because the original hashes are, themselves, hashed and not available (so you don't have a hash to work towards). And second, even if you somehow managed to get over that huge leap, raw noise won't pass Apple's review, so you have to reverse-engineer a new image that looks like CSAM, to match a hash you don't have. Big leaps required.

Technical nitpick: the hashes are not hashed (again), but rather cryptographically blinded. This is not reversible, so seems like a hash, but computations can be done on it which result can in some cases be seen by the server. This is actually all very clever, and described in a paper by Apple, linked from the technical summary article.

The collisions are supposedly reviewed, my concern is that the process for photodna isn't going to always be the same, and we don't actually have any knowledge as to whether their claims are true. Eventually they will phase out human intervention and replace with gameable AI. 3 letter agencies don't need to review the actual images, they can easily get federal warrants based on some numbers.

Apple is content with putting in backdoors. Theres collusion behind the scenes here, they know it's bad, but hey, it's apple, it's ran by manipulators and liars and forms it's own cult.

One thing about this is that we are farther and farther away from "if there is evidence for a crime a jury can understand it and rationally decide what do with it" and we are getting closer to "if this lightbulb is glowing the machine says they are guilty, so better trust us".

This kind of stuff should not be evidence, if anything it should be a indicator where to look.

In what case do you think the hash will be introduced as evidence, where the actual image being hashed, which is a grey blob, cannot?

> I’ve dumped the entire iOS ecosystem in the last week.

Not to go off topic from your main point, but what did you move to?

I'm not the one you responded to, but did something similar and feel like sharing:

- Desktop: Manjaro Gnome, for that amazing macOS-like desktop. It even does the 3 finger swipe up to see all your apps with Apple's Touchpad.

- Phone: OnePlus 8T with microG variant of LineageOS (alternatives: Pixel line-up), because that allows me to still receive push notifications. I have over 40 apps and only found 1 that didn't work so far (which is Uber Eats, because they seem to require Google Advertisement ID). I pushed a modified Google Camera app to it, so my camera is better supported. I think only 3 out of 4 cameras are working, but I don't care.

- Watch: Amazfit GTR 2e with the official app. Alternatively it should work with Gadgetbridge if you don't want to use the offical app ("Zepp"). Amazfit GTR 2 is a better option if you want it to have WiFi and want to store music on it.

Thanks for sharing. Time to start compiling these lists.

Linux, dumbphone, DSLR. This was the final straw on a planned exit to be honest.

That's interesting, I may just do the same thing as well - the camera is the largest thing of why I want to be on an phone.

I may just go LineageOS.

If you have a Pixel, or are willing to buy one, CalyxOS [1] may be worth a look. It's a privacy-focused ROM that still integrates microG, so it's compatible with most apps (unlike the other popular privacy-focused ROM GrapheneOS [2] which doesn't support microG).

The big advantage to CalyxOS or GrapheneOS versus Lineage is they support re-locking the bootloader, which means that verified boot still works - important if your phone gets swiped.

[1] https://calyxos.org/ [2] https://grapheneos.org/

> It's a privacy-focused ROM that still integrates microG, so it's compatible with most apps (unlike the other popular privacy-focused ROM GrapheneOS [2] which doesn't support microG).

Many apps are also compatible with GrapheneOS and installing https://grapheneos.org/usage#sandboxed-play-services provides broader app compatibility.

Thank you for the info, I will go buy and experiment about this.

Vindicated?

This can’t be used for a targeted attack. It’s no different from the previous false posting making this claim.

Imagine if WhatsApp and other apps added received photos to iPhone's iCloud Photos gallery by default.. wait, they do.

Imagine if Apps had to ask permission before they could save photos…

Imagine if the local photo storage is not the same as iCloud Photo Library…

Imagine if anyone who cared could simply switch off iCloud Photo Library…

Imagine if our cognitive capability wasn’t exploited.

> Within a few years your entire frame buffer and camera will be working against you full time.

This. This is hard enough to explain but with all the PR around whether the feature is good or bad it's even more difficult. Putting this functionality on device is equivalent to modifying the hardware directly, because software is eating the world.

Is it theoretically possible that photo filter effect could raise the false positive rate to make the 1 / a trillion number off by several order of magnitude?

Could be either malicious or accidental (less likely presumably).

You feel vindicated because someone made one image look like another?

That's just a checkbox that indicates further action is required. All hashing algorithms have potential collisions.

Now, the hashes' database is not public, AFAIK it is in the hands of the FBI.

What exactly is the targeted attack? To get a bunch of someone's random looking photos reviewed by a human?

Your account has been breaking the site guidelines by using HN for political battle and also with personal attacks, like here and https://news.ycombinator.com/item?id=27686351. We ban such accounts, so please read and follow the rules if you want to keep posting here: https://news.ycombinator.com/newsguidelines.html.

(In case anyone is wondering, we don't enforce the rules based on political positions and I have no idea what this account's political positions actually are. It's faster for moderation to skim comments without looking at that.)

Working against you?

This technology doesn't make it even an ounce easier for Apple, or some evil three letter agency, to spy on you. If they want to see if you're a Trump supporter, anti-vaxxer or whatever, they already have access to all your photos and emails, on device. They even have OCR and classification of your photos. An intern could add a spying function in an afternoon.

if OCR(photo[i]).containsString("MAGA") { reportUser() }

It didn't even last a week.

Introducing the people asking us to trust them with the responsibility of mass, automated crime accusations.

Why do you think somebody will accuse you of a crime because you have a photo of a grey blob?

The real world isn't as stupid as the computer one. The justice system is not deterministic and automatic. Nobody is going to look at this grey blob and go "welp, we have no choice but to throw you in prison forever"

The real world and the computer world intersect. This is precisely typified by what is being discussed, surely?

Apple is trying to automate and computerise a process that was not automated previously, apply it to a huge number of people, and with disastrous potential consequences should their wonderful design be found lacking.

And within days, they have already utterly failed to provide one of their own self-stated and incredibly obvious requirements. What else could possibly go wrong?

Further, if you think that because the first pre-image failure found (in days...) was a grey blob, that this means all future possible cultivated collisions will only ever be grey blobs, well - good luck with that.

The process of making accusations and arrests is not automatic.

The Apple system sends flagged photos to reviewers, and if the reviewers find them suspicious, they send them to the poor souls at NCMEC, who will compare the flagged photo with the original illegal photo that's supposedly a match, and inform law enforcement if they are in fact a match.

Nobody will get cops at their door because somebody sent them a grey blob image.

The process that's being "automated" is merely an automatic flag that initiates a several-step process of human review. Apple isn't rolling out robocops.

Seriously, what are the "disasterous consequences" that you envision? What is the sequence of events where a hash collision leads to any inconvenience whatsoever for a user?

What part of "they already totally fucked up how their own process is supposed to work" don't you understand?

What part of it is fucked up? They never promised that their hashing algorithm was uncollidable. The process is specifically designed to be tolerant of hash collisions (and in fact wouldn't work otherwise, because the system needs to ignore small differences between copies of the illegal images, like one-pixel edits or color temperature differences or photocopies).

There are multiple stages of human review in the process and a high threshold for activation of the proccess because collisions are inevitable. The fact that collisions exist doesn't impact the safety of the product whatsoever.

I ask again, what is the sequence of events where a hash collision from a benign image can lead to any consequence at all for the user?

"Apple said that there is a one in one trillion chance of a false positive"

https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni...

Versus:

https://blog.roboflow.com/nerualhash-collision/

Along with:

https://github.com/anishathalye/neural-hash-collider

Demonstrates blatant incompetence on the part of Apple's design. At this point, trust is broken. That's the end of credibility regarding any other claim made about the security or base function of the system.

As for a simple example of one of many potential attack vectors (search HN for "neuralhash" and read the comments for countless examples), here is a basic attack:

As Apple's hash algorithm is entirely broken and useless, a malicious actor can easily craft a NSFW, but otherwise legal image, that has a hash matching one in the CSAM database, that is visually ambiguous in that context (for example, a close-up). Targeted attacks could be launched on unsuspecting victims as simply as sending them an image.

If their app is configured to auto-save to iCloud - as some are by default, and as many are configurable to be - the image will then get automatically flagged as a match, and may well potentially pass a human review - because the image may indeed look just like CP - and an innocent person, who may not even be aware of the presence of the image in their iCloud library at this point, may get visited by the police.

That in itself is bad enough.

However it gets worse - do bear in mind that "the police" is not an equivalent concept across nations, yet iPhones are ubiquitously used across the world. In certain countries, a "visit from the police" over such automatically-flagged content could well result in presumption of guilt to the point of immediately administered punishment and a destroyed innocent life.

In China for example, conviction rates are routinely 99.99% - being accused of a crime there is equivalent to being convicted, and that's just one major country containing well over 100 million iphone users.

Apple is demonstrating not only gross incompetence in this design of this "system", but vagrant disregard for human rights, along with utterly destroying their own long-cultivated pro-privacy stance.

It's a disaster for them and their customers in every possible way it could be.

Apple was talking about the chance of accidental false positives.

There is a 1 in 4 billion chance that if I send a message to an IP address, it will be yours. The fact that I can know your IP address and send you a message intentionally does not make the previous statement false.

> As Apple's hash algorithm is entirely broken and useless, a malicious actor can easily craft a NSFW, but otherwise legal image, that has a hash matching one in the CSAM database, that is visually ambiguous in that context (for example, a close-up). Targeted attacks could be launched on unsuspecting victims as simply as sending them an image.

Everybody keeps talking about this scenario as if it is any different whatsoever from sending people actual regular old-fashioned CSAM, which A: already happens, B: requires less effort on the part of the attacker, and C: isn't reliant on the recipient uploading them to iCloud, triggering several layers of filtering and detection, which may or may not result in law enforcement interest depending on your forgery's ability to fool professional investigators at NCMEC who are comparing it alongside the original.

The attack you're describing is less effective and more difficult than existing ones.

No, the attacker is committing a crime with the attack you describe, but has plausible deniability that they are not with the Apple-nonsense enabled attack vector.

This is a huge difference as not only does Apple facilitate a gain-of-function in terms of potential detriment to the victim of an attack, the newly-enabled vector also indemnifies the attacker.

The effort required to come up with a hash collision in SHA256 is something on the order of the power output of a trillion suns for a billion years (or some other meaninglessly-prohibitively-large amount). Despite such metrics and off-the-shelf solutions being readily attainable within this field Apple has implemented a system that has resulted in a collision within days, followed by a free automated collision generation tool.

This demonstrates absolute gross incompetence in design, and permanently eliminates any trust any sensible individual can ever have in the system despite your protestations.

There is simply nothing you can say to convince any sane person otherwise, the situation has been factually described and it is what it is.

It can't. No actions are taken on hashes alone. The procedure is, if an account uploads some number of images with matching hashes, those images are verified by a human.

This can attack that system itself, though, by overloading those humans with too much work looking at random noise, but that requires quite a large organised effort. It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography.

"Verified by a human" - and that human will be an overworked, underpaid, overseas subcontractor who may well have an incentive to mash the "Confirm match" button from time to time to improve his performance.

Beyond that, are we seriously going to ignore the fact that eventually this system will share the private photos of someone's naked kid with some random subcontractor? How is that even remotely OK? Or that it will find and share actual CSAM with said subcontractor?

It only shares "visual derivatives" of images whose NeuralHash match the NeuralHash of known CSAM (either by being the same image ("perceptually") or a collision).

That to me just sounds like weasel words to avoid having to say that it shares images. Let's not beat about the bush, the "visual derivative" has to be good enough to identify what's going on in it for the manual confirmation.

Are you actually arguing in good faith here at all? Because I can't see how a "visual derivative" that's nevertheless good enough for manual confirmation is any better than the source image?

> Are you actually arguing in good faith here at all?

Are you? Because you just seemed to claim that it could match against innocent pictures of your naked children, but this tells me that you don’t understand that this system looks for known pictures, not for something that looks like naked children.

Edit: if you do, apologies, but then I’d say that Apple has suggested that it’s a low resolution version of the picture. This should be contrasted with server side scanning, where the server accesses all pictures fully.

This system does not use ML to find new CSAM images. It only checks for ones already in a known database. Your pictures of kids in the bathtub are not on the list.

What is show to the reviewer is a "visual derivative" which hasn't been clearly defined. A thumbnail image? Something with a censored section? We don't really know.

Yes I'm aware that it checks against a known database but clearly there can be collisions. So eventually it will share someone's private images.

It would need to have 30 collisions before anything even took place, which realistically isn't going to happen.

Most cameras nowadays can easily take a burst of 30 visually very similar images in a second.

Lots of people leave their cameras in burst mode.

Or rather, it will share some gray blob apparently.

What is the point of sharing a gray blob? How is that going to prove anything?

I think the gray blob refers to engineered hash collisions, like the example in the article link.

The "verified by a human" is only the the last step of the process for Apple. After that, it's given to NCMEC for review, then it's reported to authorities. The Apple subcontractor isn't the only one verifying things here. What you're saying is not realistic.

I find it very hard to believe that Apple will put literal future of iPhone (just imagine the amount of bad press if one false negative comes out of review process and is reported to the authorities) to "overworked, underpaid overseas subcontractor".

Apple is greedy, ruthless, tone-deaf machine, but I don't think they're stupid.

That’s already how app reviews work.

I think rejecting one of million apps and angering one developer is a bit different than reporting someone about a child pornography/abuse, ruining their life and making a global scandal that can put future iPhone sales in jeopardy.

Barely anyone ever cared about privacy. Convenience, no matter how insignificant, always wins. I will be surprised if this is the last straw that finally gets the masses to care about privacy. In reality there might be a few scandals but at some point no one will care anymore and Apple will remain popular.

> It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography.

I've never personally seen or looked for any images like this, but if they weren't already proliferating online and widely available to criminals, why would we need to build an elaborate client-side scanning system to detect and report people who have copies of them?

Do you have access to darknet child pornography trading networks? I don't.

OK, now I'm curious. How does darknet stuff actually work?

Is there some darknet search engine that you access over Tor and type whatever illegal thing you seek such as "child porn" or "stolen credit cards" or "heroin" or "money laundering" into and it points you to providers of those illegal goods and services reachable over some reasonably secure channel?

Or is it one of those things where someone already using a particular child porn or stolen card or money laundering or heroin selling site has to put you in contact with them?

Hmm. Tor onion addresses are generally super long and have a large random element to them, such as

facebookwkhpilnemxj7asaniu7vnjj.onion

where the "Facebook" part is there for convenience and may likely not be present for actual illegal addresses. As such, you probably won't be able to find a CP website unless you actively brute force the entire onion address space.

I don't know how the sharing actually happens, but I don't think a real search engine for the dark net exists. There are websites that try (Ahlia, I think?) but they are super limited in scope.

I suspect addresses for drugs and CP are shared person-to-person.

dark.fail indexes darknet markets, but all the ones I've seen ban CSAM

Agreed. This attack still has consequences for people but they will not run into any legal trouble as far as I'm aware.

I doubt someone who is able/wanting to attack such a system would be unwilling to own some of the material themselves.

That assumes that the human review process is competent, your own images aren’t poisoned in some way (consider your own kids in the bath with some noise added) etc.

In the mean time they lock your account which means your entire digital life stops dead until their review process is done.

No way do I accept any of this.

Also the hashes are on the device I understand and it’s not going to be that difficult to extract them.

You:

> the hashes are on the device I understand and it’s not going to be that difficult to extract them.

----

A Review of the Cryptography Behind the Apple PSI System, Benny Pinkas, Dept. of Computer Science, Bar-Ilan University:

> Do users learn the CSAM database? No user receives any CSAM photo, not even in encrypted form. Users receive a data structure of blinded fingerprints of photos in the CSAM database. Users cannot recover these fingerprints and therefore cannot use them to identify which photos are in the CSAM database.

https://www.apple.com/child-safety/pdf/Technical_Assessment_...

----

The Apple PSI Protocol, Mihir Bellare, Department of Computer Science and Engineering University of California, San Diego:

> users do not learn the contents of the CSAM database.

https://www.apple.com/child-safety/pdf/Technical_Assessment_...

----

A Concrete-Security Analysis of the Apple PSI Protocol, Mihir Bellare, Department of Computer Science and Engineering University of California, San Diego:

> the database of CSAM photos should not be made public or become known to the user. Apple has found a way to detect and report CSAM offenders while respecting these privacy constraints.

https://www.apple.com/child-safety/pdf/Alternative_Security_...

So fundamentally there’s a lot of mechanism around keeping secrets but the source material is available. Hmm.

https://youtu.be/eU2Or5rCN_Y

The human reviewer would be able to check against the exact image that generated the hash in the first place. Taking another completely unrelated image and perturbing it would be immediately obvious.

So there’s an office somewhere with computers full of illegal child porn that people are staring at and comparing your photos to?

There’s some irony in that.

Yes. That is called NCMEC in the US and it is a core aspect of how this whole process works.

If you don’t understand the details of this, I’ll recommend this podcast episode which sums it up and discusses the implications https://atp.fm/443

I am aware of the details. The episode of Brass Eye comes into context here, the relevant clip being as follows, showing exactly the issues of competence.

https://youtu.be/_U-7L1tmBAo

I don’t have much of an opinion here except that it is silly to write “I am aware of the details” when someone gives you a helpful explanation and you are obviously not aware of the details, as you had just asked a question betraying.

I’m sorry, what is your point? I legitimately don’t understand what you mean.

> In the mean time they lock your account which means your entire digital life stops dead until their review process is done.

Do you have a source for this, or are you just making things up?

the blacklisted hashes are openly given to thousands of companies by a foundation called the National Center for Missing & Exploited Children, in both PhotoDNA and MD5 versions of the hashes.

Yes, I can't give you an open link, but I imagine at least hundreds of thousands of people have access to download them. Some companies with better security than others

It only takes 1 guy to sell it to an NSO-type company, who would then use it to target attacks for $$$

Where would that $$$ come from? Like I said, the only attack possible is to overload the human reviewers, and even that is difficult to pull off except as a coordinated effort by lots of people.