It is a very tenuous theory that this will only end at CSAM with alleged trillion to 1 odds. It is unclear how they justify stopping there - all laws are roughly equal in the eyes of government, why shouldn't an iPhone enforce them all? This incident shows that Apple is comfortable with the idea of using my phone to get me imprisoned:

1. I don't want to financially support anyone going through my private things in conjunction with what is basically the police, looking for reasons to imprison me. Reasonable suspicion first, thankyou very much.

2. I don't trust them to use this in a politically neutral way. This is too much power.

3. I don't trust them to manage the false positive rate. Everyone knows how reliable software engineers are when they claim a system does something. At heart they aren't 1 in a trillion people.

This is a great time to be outraged. I can't really do much about what they do on their servers, but I can certainly get antsy about what happens on my phone.

It's not like this is some new fancy technology no one could do before. There has never been any technical barriers to scanning your phone for insert-reason as part of the OS. The slippery slope argument could be made from the first smart phone. The government could always pressure any company to put a backdoor in any OS. Apple hasn't done this (to our knowledge) so far and this new CSAM scanning feature doesn't remove any non-existing barriers to that end.

They just gave a talk at the USENIX Security Symposium on how this works and the safeguards put in place. The scanning functionality is part of the icloud photo upload module. Security researchers can determine if they change the scanning algorithm in future updates. I don't know what else you can hope for.

Going back to the slipper slope argument, you could say nothing stops them (or Google on Android) from uploading your passcode and share it with X. It's all just software in the end that they write that powers the lock screen and security checks.

> Going back to the slipper slope argument, you could say nothing stops them (or Google on Android) from uploading your passcode and share it with X. It's all just software in the end that they write that powers the lock screen and security checks.

If they start claiming they do that then I'll get angry about that too. You'll notice that only the fringe is accusing Apple of being liars or acting in bad faith here. Apple are pretty up front about what they do.

The problem with this plan is they're claiming they are only going to do selective law enforcement (only 1 US law, only things that are strongly supported by consensus and even then only sometimes). That isn't a position with a reasonable foundation, they are going to change their mind. I want them to change it in the "we don't snoop on people's phones" direction rather than open season.

A possible workaround - but not remotely one I expect them to do - is for them to amend their own terms of use to forbid themselves from expanding this feature without explicit opt-in consent, forbid themselves from making any expanded version mandatory to continue using the device with their software, contractually agree to liquidated damages in case of breach equal to three times the greater of original phone MSRP or actual purchase price plus included taxes/fees and inflation and interest and attorney/court fees, and remove their right to amend this provision without that same non-mandatory opt-in consent.

Again, I don't expect them to do this. But they could add a useful level of trust if they were really motivated to do so.

Then they could just change their ToS when they wanted to add more.

Not true. If they had previously adopted my suggestion in the old ToS, that itself would either be a breach of that ToS (leading to liability for prohibitively large contractual damages) or would be an invalid change. I suggested that they explicitly remove their right to make this kind of change unilaterally. Terms of Service don't have to allow their authors an unfettered unilateral right of amendment.

And what’s the recourse if a company breaks their own ToS? Also no company will give up any rights as they have no real way in knowing if the gov will come force more images to be added. What then?

The main recourse in practice with the arrangement I described would be a class action lawsuit alleging on breach of contract. I'm quite sure courts would have no problem with the lawsuit, assuming there were no arbitration clause or class action waiver.

As for no company giving up rights, I did say I didn't expect Apple to actually do this, only that they could. Though, companies do actually make binding contractual promises all the time. It's just that they don't tend to do it with consumers or small businesses who have no negotiating leverage and/or no desire to insist on those promises.

Regarding your hypothetical, anything that's actually legally mandated by the government generally overrides contract law since the courts won't enforce illegal contracts and illegality is a defense to breach of contract. They can always do what the law actually requires.

But we're not talking about the government mandating things, since both the iOS 14 behavior and the iOS 15 behavior comply with the law.

> Regarding your hypothetical, anything that's actually legally mandated by the government generally overrides contract law

Which means this action is just a charade, as I pointed out. Because fundamentally it doesn't really matter. The gov is going to do what they do, and Apple is not going to create a situation in which an external group can determine how or if they can be sued in court based on how overreaching they feel that day.

Also go check how many arbitration clauses exist in your ToS, just as an exercise. I'm all for banning arbitration clauses and allowing consumers recourse in actual courts. I don't believe we should have companies sign their death sentence prior to doing business, this raises the barrier to entry and results in even less competitors who have to do the same.

> I don't know what else you can hope for.

I could hope for being able to trust that an intensely personal device such as a smartphone isn't something that I have to be constantly suspicious of.

Admittedly, that ship sailed a long time ago, but I could still hope.

That ship was never in port. Apple control the hardware and software and always have.

You being able to trust them is a personal choice. It seems people are happy to trust companies that don’t talk about what they do with their data rather than the ones that do. Which, while understandable seems counter intuitive.

I keep hearing this argument and I don't understand it. The entire world would know within about 30 minutes if Apple ever pushed some malware update that did an end run around security and started grabbing files off people's hard drives. I don't trust them, I know they don't do that - and we would know pretty quickly if they did. Now they're saying they will.

Could they write and deploy something overnight that hoovered up everyone's data? Maybe on iOS, less so on OS X, but now they're going to ship with that capability. I don't understand how some people can't see the difference between "they could always push some nasty update," versus literally shipping hardware and software with a backdoor.

> The entire world would know within about 30 minutes if Apple ever pushed some malware update that did an end run around security and started grabbing files off people's hard drives

But it would be very hard to spot new weights of the neural network and a new list of target hashes. These are pretty much guaranteed to change regularly as they retrain the embedding network and/or change the list of known target images. So it will be very hard if not impossible to see what they're searching for. That latest update could just add the ability to recognize CSAM pictures that had meme texts added to them. Or it could change the embeddings and target list to spot unlicensed posting of copyrighted still frames from movies. Or it could now retrieve any picture of people in police uniform. No way to know if you don't have a hunch and a targeted picture you want to test with.

> I keep hearing this argument and I don't understand it. The entire world would know within about 30 minutes if Apple ever pushed some malware update that did an end run around security and started grabbing files off people's hard drives. I don't trust them, I know they don't do that - and we would know pretty quickly if they did.

Do you know this for certain about any vendor? If a company the size of Apple were pushing the same update to everybody, then it would likely be known about by the world pretty quickly. But... if a targeted signed update is sent to a handful of selected devices, that's harder for the world to find out about. It's risky, but it's definitely technically possible.

I'd like to respond to all three responses: We have the ability to monitor upstream traffic. If Apple takes away that ability to see what's being pushed on the device, we can look for suspicious traffic on the router or network level. I personally don't let any of my devices communicate with known Apple, Facebook, Microsoft or Adobe IP addresses, so it'll be interesting if they suddenly start popping up new addresses in order to do so.

People do know the difference. The point of disagreement is in what you said firtst. You say that you know they don't currently ship with a backdoor, while others say you cannot possibly know that.

Quit talking down to people who don’t want Apple messing with their private party. Just accept that they have different opinions that you do.

> I don't know what else you can hope for.

You cannot be serious.

> I don't trust them to manage the false positive rate.

Let’s compare:

— Apple’s market cap is more than twice of Facebook’s.

— Apple’s user base is less than half of Facebook’s. Even fewer use iCloud Photos (and much fewer upload more than the free 5 GB, shared across photos and other content).

— Facebook is entirely made of UGC with no storage limit for a given user, and their moderators have to review every bit of content to ensure the gore is away from advertising targets’ eyeballs (and, indeed, report CSAM). Apple only needs to review images that trigger multiple hash matches occurring within the same account.

Considering the above, my intuition is that Apple’s NeuralHash would have to be completely broken for a company this size to not be able to afford enough moderators to manage the false positive rate. I sincerely doubt they are so inept, and I’d be willing to live with the potentiality of an Apple employee peeking at a photo of mine once a year (me using Apple tech already implies I trust them enough, as I’d never be able to personally verify every privacy claim they make).

What is worth screaming about, and what is eroding my trust, is the fact that since 2019 Apple’s ToS (quietly changed, presumably, to accommodate this feature currently in the news) give them carte blanche for pre-screening any content that they deem potentially illegal. Unless they tighten up that phrasing to limit it to CSAM only, they’re allowing it to be used as a political prosecution tool.

Edit: Factual error, iCloud Photos does have a free tier.

Your argument here seems to be that Apple is adopting privacy policies that can be favourably compared to Facebook, and have taken a step closer to how Facebook handles data. I don't disagree at all. That is the problem I see here.

Anyone trusting Facebook with a level of access to their life comparable to a mobile phone is foolish. There is no way I would pay money for Facebook to control my phone.

I only explained why I think the parent comment’s doubt about Apple’s capability to screen false positives is most likely unfounded.

Capability != willingness

If it costs them too much money to have too many people personally looking over every single positive hit to make sure it’s not false, then they’re not going to do it even if they technically could.

Yes, and I tried to summarize why I think the cost of hiring enough people for this particular job is extremely unlikely to be more than a rounding error for Apple.

What are they achieving? Let's say there is a pedo who has cp on his iPhone. He is scared that apple might be after him but he also sees how this scanning only applies to cloud scanning so apple also gives this pedo option to not getting caught so what is the point ?

I live in a place where the government arrests you for having a VPN installed on the phone and labels you a terrorist outright. My phone is checking with physical frisking on the roadside and content critical of state gets automatic manhandling and trip to the police station where I am treated as a criminal.

How I see this as a problem not because I am not going to buy an iPhone in future, but because such ideology would be made normal. That is what is scaring me

Exactly.

The technical implementation is trying to hide the fact that the design mojo of this system is an actual Backdoor for governments of the world to oppress, censure and do whatever they like.

The technical implementation is optimized towards minimizing the cost for Apple. That's why they scan on the device. And this decision is made with knowledge that in the near future "scanning" will be not limited only to hashes. Scanning and processing will be required for $Some_Cool_Functionality to work.

Like GMail?

Gmail is on the Google servers. Did you pay for Gmail? Did Gmail has access to your computer by default? So on.. What analogy are you making? When Gmail advertised "What is on your Gmail, stays on your Gmail"? :)

No, no analogy. GMail scans every email and every attachment. Server side because - of course - it’s an online service. It’s main feature is analyzing and indexing so it’s a direct example of a killer feature based on content analysis. Even spam filters are.

The same slippery slope applies there.

According to Sir Pratchett: 1 in a million occurrences happen 99% of the time.

* Sir Terry Pratchett

It's Sir <firstname> or Sir <firstname> <lastname>.

I thought this was interesting so I looked up the naming conventions for knights. For the curious here’s some info:

https://www.geni.com/projects/Naming-Conventions-for-Knights...

It's also using the resources of your phone that you have paid for to scan and police your own content, much like AV software. I assume the hashes are either uploaded or stored on the phone.

There was zero confusion. Apple's new feature was universally rejected and if it cone to market will cause a severe loss of market share.

Severe loss of marketshare? It wouldn't even register.

I don't like the feature. Putting this on the client device is dubious and should never have made it past the brainstorming stage.

Having said that, technology companies, big and small, are bound in the US to do this. By law. If anything Apple was by far the laggard of the bunch (with reporting counts magnitudes lower than peers, despite a larger customer base). As I said in another comment, no company can protect you from your government.

Much has been made about it being on device, which while a serious optics issue...the hot takes being given on here are manifestly absurd. Like, literally the company that holds all of your data, all of your passwords, all of your info and you need to invent slippery slopes to imagine up what they "might" do?

If they want to have their way with your data, they could have been doing it for decades.

They should never have announced two very different systems at the same time. Contrary to some of the insincere claims given in this very thread, there is massive disinformation and confusion about them. In the end I feel like 98% of the "the end is nigh!" comments are by long time Apple detractors who just see this glorious opening.

And while I still hope that Apple says "Mea culpa, we're just going to scan on the ingress to iCloud Photos", whatever they do in a month this is going to be completely forgotten.

> Having said that, technology companies, big and small, are bound in the US to do this. By law.

Law requires reporting CSAM. They are not required to scan for it.

US law makes them responsible for what they host, to a reasonable, discretionary degree. Literally *every* technology company that I know of scans for CSAM for this reason. Apple is only implementing this for photos that go to iCloud Photos for this reason.

They are surely not required to scan client devices, and that was a foolishly, ill-considered plan (that I still would wager they will abandon), however they absolutely must scan iCloud Photos unless they were technically incapable of doing this. US law doesn't say "you go to jail if you don't", it says "you face enormous liability if you don't".

I have dedicated and virtual servers at OVH and Hetzner and other companies. I'm quite confident they don't scan for anything on the hard drives. One of the selling points of Scaleway.com's C14 backup product (previous incarnation, don't know about current one) was that the storage was encrypted and that they would delete their copy of the decryption key if you checked a box. If you chose that option, you could not restore your backup without re-entering the key (block of hex digits that you had to copy on making the backup). It wouldn't surprise me if that resulted in the obvious sob stories about lost keys.

Those are not US companies but I've never heard of AWS, DO, etc. scanning people's storage either.

AWS offers encrypted backups, and afaik there's no scanning of anything you put in a bucket, but why trust them? Just encrypt your own backups before uploading them, and keep your own keys. In other words, your storage provider needn't and shouldn't be the purveyor of your backup software.

Except with an iPhone, you don't have a choice.

> but I've never heard of AWS, DO, etc. scanning people's storage either.

Pre-Snowden, that was insourced by the NSA.

Tech folks drastically overestimate how much the average person cares about privacy.

Plenty of people believe that the Facebook and Instagram apps are recording audio 24/7 and target you ads based on the speech the apps hear. That doesn't stop people from using the apps.

A few years ago some of the most famous people in their world had their iClouds accounts hacked and had their naked photos leaked. That is a lot of people's worst fear. People literally commit suicide over this sort of thing. It didn't hurt the iPhone's market share.

People largely don't care.

IMO, people largely DO care. If you ask the average Joe - is online privacy important to you, the majority would obviously say yes. Why then don't they stop using FaceCrook?

1. The penalty of social ostracism due to the network effect. This is a severe punishment to most humans - particularly in today's socially disconnected world. Without these apps, a large # of people would not have any contact with much of their social circle - including family.

2. Learned helplessness. I think that many people have just given up. Even if they knew how to fight for their privacy, they see time and time again that money always wins.

A better way of saying it would be that they just don't care that much. Most people care in theory about taking care of the environment, but that doesn't mean that they're willing to tolerate even 1c/gal higher fuel taxes. Most people care in theory about taking care of the poor but balk at the personal economic consequences of doing so. People have different priorities and when one consistently loses out to the others that means that they largely DON'T care about it.

> Tech folks drastically overestimate how much the average person cares about privacy.

This is old thinking. In the 90s and 00s it was miraculous to trade in a little privacy for some awesome free service on the internet. Google, Facebook and others became huge, and everyone has been influenced / manipulated by them and people are getting tired of having a family dinner conversation next to their smart speaker and seeing ads for six weeks for Depends diapers because someone told a bad joke at the dinner table.

I’d argue that “privacy” is just a significant trigger argument for most people, but not felt as important per se outside of the context of being afraid of the social consequences of some behaviors being exposed.

Take all the paranoia and “muh privacy” around contact tracing or the European Green Pass: It’s as anonymous as it can be, yet millions of people argue against them across the political chasms that separate them. So yes: unless it becomes a divisive topic (which would be undesirable), this CSAM scanning will go away with the news cycle.

About your comment on celebrities’ “sex-tapes”. I guess it takes a certain kind of extroversion and (positive) narcissism to be one, and they’re expected to be gossiped about or to show their bodies in movies or photos so I don’t think those victims blinked that much - some like P. Hilton probably manufactured some porn to ride the wave. On the other hand, revenge porn did drive ordinary people to suicide because in this case the victims weren’t prepared or expected to let the public to see that.

If privacy was a selling point to buy Apple, then it will be a selling point to buy a competitor.

Who knows how much it will matter in the end? But it is hard to argue it doesn't matter.

> Having said that, technology companies, big and small, are bound in the US to do this. By law.

Other companies scan their servers instead. And what law banned E2E encryption?

This actually enables Apple to apply end-to-end encryption, while still complying with US CSAM laws. Users' content is scanned for CSAM, and is encrypted before leaving their phone. Whether one considers E2E encryption worth it if the content is still scanned before being encrypted is an exercise left to the reader.

This potentially means all of iCloud, not just photos, could start to use E2E encryption as well - which is fantastic.

>This potentially means all of iCloud, not just photos, could start to use E2E encryption as well - which is fantastic.

They have to scan things that are not photos. What if the bad guys just zip their photos and upload that?

There also needs to be a solution to CSAM uploaded before NCMEC had a chance to tag it, especially to cases where the bad guys uploaded their CSAM and deleted it from their iPhone. What happens then, the bad guys get E2E and nobody can find them? There has to be a technical solution in mind for this, and everything I can think of has implications (Let the iPhone store hashes of deleted images? Would it be enough to scan also during download?)

IMHO, any serious attempt to find CSAM using Apple's client-side approach requires more scanning, and Apple not being forward on that makes me trust them less. Also, the moment they expand the scanning, we should think carefully if there actually are any privacy benefits.

Leaking information to Apple makes it not E2E.

What law do you think banned real E2E encryption?

There's no leak of content. None of your content ever gets sent to Apple...ever.

"safety vouchers, which contain a visual derivative of the image, such as a low-resolution version"[1]

And I said information. The hash matching is information.

[1] https://www.apple.com/child-safety/pdf/Security_Threat_Model...

You edited your response because I specifically used the word "content" because that's what you used...

I did not. The person I replied to said content. Not me.

Which law specifically says that user data has to be scanned, breaking encryption if necessary? Everything I've seen says that they have to report what they find, but are under no obligation to actively look for it.

So if we submit to having all our content scanned by them they'll even give us E2EE on iCloud? I guess they really do care about our privacy.

I don’t care about the cloud. I don’t want anyone scanning my device.

Only photos that get sent to icloud are "scanned"

For now.

What law?

> are bound in the US to do this. By law

I keep seeing this asserted, but there is never any legal citation. What exactly compels a software company to make their software product scan for CSAM?

I get that a service provider like cloud storage may need to scan what they themselves are storing to avoid possessing such material themselves. And iCloud could scan uploaded blobs all day to fulfill their legal department's recommendation.

But what exactly compels a software developer to include a content scanning function in code they distribute? And does this requirement also apply to the authors of rclone?

What distinction are you making with what I stated? What disagreement is there?

Apple only scans photos being uploaded to iCloud photos. Google scans. Facebook scans. Microsoft scans. Even tiny image hosting sites scan.

Apple decided to implement this functionality on device, but they could as easily (with much less fanfare and dissent) have placed it on the ingress to iCloud.

The distinction is between providing services and providing software.

When iCloud scans stored files like Google Drive, nobody complains. It's understandable that risk adverse legal departments have come to the conclusion that doing such things is necessary, to avoid a company being in possession of trivially-discoverable CSAM. And from an individual security perspective, you should consider everything you upload unencrypted to be the subject of similar analysis.

If iOS were to encrypt all files before uploading, making any iCloud scanning mostly pointless, Apple would still not be running afoul of any law. Apple is making general software for end users, and encrypting files to upload would be doing the basic user diligence I alluded to. If users end up using the software to do bad things, those specific users are liable and not Apple.

As far as I am aware, there is no legal requirement for a software developer to modify their software to perform scanning of content it will process while being run by other people. But this is what is implied when you say that Apple is forced to do this by law.

Furthermore if this development was driven by iCloud worrying about legal liability from the combined system, then iCloud should be spun out into a separate company that cannot affect the development of the iOS software.

This is the closest explanation I’ve found: https://www.bbc.com/news/technology-51207744

Yeah, I dislike Apple and want them to have less marketshare, but I doubt they will even lose 1% of revenue over this.

> in a month this is going to be completely forgotten.

I'm tossing the mac and iphone because my phone is mine.

> If they want to have their way with your data, they could have been doing it for decades

Yep. They just altered that deal. (Darth Vader quote deleted)

> Severe loss of marketshare? It wouldn't even register.

Disagree.

Something this ludicrously intrusive, over-reaching, reckless and trust-destroying is a dealbreaker for exactly the demographic of technology enthusiasts who influence others purchasing. It may not be instantaneous, but it would certainly propagate.

It's anecdotal, but I know zero people who are NOT reconsidering even lifelong loyalty to Apple over this.

It's just too crazy big a wrong, it's like the company just had a nuke go off inside it and is trying to pretend nothing happened.

They may as well have announced a partnership with Trump to put MAGA engravings on all future products, and then in the ensuing furore say they "regret the confusion", whilst carrying on with it regardless.

Though I suppose in that scenario they'd at least be targeting a significant market.

The same can't be said for the size of "sign me up for software-automated police raids" market. A market whose naiveity-induced initial "size" would rapidly shrink after the first few innocents went down, as they absolutely would.

I mean depending on the targets chat software preferences, could a malicious actor potentially ruin an Apple iOS users life just by sending them an image?

There are many troubling scenarios one could imagine. In fact, there is nothing but troubling scenarios.

It's not a can of worms so much as a wormhole, blasting an endless stream of worms at the speed of light.

I'm left not just questioning Apple's leadership, but - honestly - their mental faculties. Really.

No innocent person wants to walk around with an automated snitch in their pocket - I mean, hacking? Bugs? Oversights? Just the overall preponderance of fear that every millisecond you walk around with this thing, it - and its parent corporation and all the depersonalised machinations that go along with it - could be busy organising a blithely mistaken police raid on your family?

Say, because someone you've never met, sent you a message on that new chat app you forgot you installed, that autosaves all media to your iCloud? Or because someone stole the spare phone you keep in a drawer at work, used it for God-knows what and you didn't even notice it was gone? Or, maybe your teenage son got sent something from his teenage girlfriend who unbeknownst to any of them had her phones images uploaded and subsequently catalogued and flagged? Or any number of other entirely plausible scenarios that provide the very reason we have law enforcement protocols and procedures for reporting crimes and that are complex, nuanced and have evolved over hundreds of years and mountains of cases into a massive structure that exists primarily to protect the innocent from exactly this kind of freaking crazy shit?

And Apple expects people to pay them to carry the weight of all that around with them?

This is misinformation.

> Putting this on the client device is dubious

Putting it server side is categorically worse. Putting it in the client SDK for iCloud (architecturally speaking) rather than on cloud storage or in the OS is clearly the better correct technical choice, tying surveillance’s hands in a way server side or OS would not.

Most every client SDK routinely checks content before upload, it’s a best practice. Careful examination suggests this was engineered better than that practice.

(Note: even tech trade posts such as LWN, Stratechery, or Daring Fireball trying to write well about this need to sit down a minute and have how it actually works walked through for them, as do many in this community.)

FWIW, I agree with much of the rest of your post except the rationale for low reporting counts.

No, architecturally there's no reason to do it on the client side. You're uploading a photo to iCloud, encrypted with a key that Apple controls. The only reason to use even a little bit of battery power to do the check on the client, and to do this elaborate song-and-dance around only notifying Apple if there are a certain number of positive matches, is that it creates the (false) impression that Apple couldn't just check your photos on the server side if they wanted to.

It's basically engineering with the goal of creating a perception of privacy rather than actual privacy. I don't really care that much about Apple policing what you upload to iCloud, but this disingenuous architecture does annoy me a bit, and there's also the added insult of having a device in your pocket that by design works against its owner (reminiscent of "treacherous computing"). These problems would go away if they just did the check on the server side.

> No, architecturally there's no reason to do it on the client side.

The architectural reason is to do things the server couldn’t.

> You're uploading a photo to iCloud, encrypted with a key that Apple controls.

The architectural reason is so the server doesn’t have to be able to read the photo, and it need not be a key Apple controls.

Your first two sentences are the exact reason to do it on the client instead of on the server, such that it’s possible to have e2e encryption opaque to the server.

But it's not e2e encrypted so I don't know what point you think you're making here.

Apple gets hung up on weird stuff. Likely doing work phone side was an environmental argument. The amount of energy used to decrypt server side to create the hash was marginally larger than the amount used by the phone pre-encryption with cost of transmitting. (Bonus that the energy and CPU was paid for by the client). The result for either location is identical.

If I were to design a spying device, I would definitively put it on both the client device and the server. That is because third party apps are not required to put their contents on Apple servers. By putting it on the client, you can potentially hoover up any app’s data. Just write a background service that scans the client’s file system and track all calls to encryption APIs.

The list of apps is such a treasure trove. Signal? Clubhouse? Telegram?

https://developer.apple.com/documentation/security/complying...

I need to research what Signal does on iOS. My next canary is encryption of messaging apps other than iMessage.

> Putting it server side is categorically worse.

Why? Putting it server-side means that it's only possible to examine the images that they claim they are targeting -- those going to the cloud. That seems like a much better and more private way to do it, because it's putting the surveillance "in their house", so to speak, instead of mine.

Keeping it in the client SDK means it’s possible to implement storage the server can never see into, in a manner that can be verified by red teams.

I do think it will result in a loss of market share if it comes to pass, if, for the simple reason that Apple will likely lose the Privacy Moat.

Upcoming contenders like Purism [1] and the Pine Phone [2] will start gaining a great deal more traction from this. Other SV firms will sense business opportunity..... If merely 5% of the TAM around mobile is willing to prioritize non-spying features that would be enough to stand up very healthy businesses.

It isn't like an iPhone is very customizable, repairable, or that usable with all the App restrictions Walled-Garden stuff.

[1] https://puri.sm/products/librem-5/

[2] https://www.pine64.org/pinephone/

> Upcoming contenders like Purism [1] and the Pine Phone [2] will start gaining a great deal more traction from this.

I’ll bet you $500 to the charity of your choice that this won’t come to be. Set the terms on how you want to measure the outcome.

I don't bet - I trade and invest both early stage, and on public markets.

Since you want to have a friendly competition around "put your money where your mouth is" will look into whether or not Purism is accepting investments and what the terms are. AAPL valuations are pretty lofty right now at ~$149 a share if you'd be interested in the reverse :)

Year of the Linux ~desktop~ telephone.

> and if it cone to market will cause a severe loss of market share.

AAPL closed 90 cents short of an all time high share price today. Why isn’t the market pricing in the loss of market share?

Because stock price is based on investors reactions to quarterly reports. Any loss in market share, if seen, will come when people usually buy new devices every x years don't.

It's a hit to their brand from technically knowledgable people for sure though. When someone asks their tech friend if they should buy Apple, more people will likely say, "yabut," or "nah." We'll see if that makes a difference in a year or three.

To me, it just feels icky. I'm sick of all the spying. I've been in computers since the Commodore. The current computer world is shit because of spying. It killed any passion I had left. It seems like no last vestige of privacy remains.

Stock valuation models get updated a bit more frequently than quarterly, new information is incorporated into the share price all the time.

The person above you gave an example of confusion. I'll admit that I was confused about how this new system would work as well. So there was confusion.

As for market share, this might barely register on people's radar outside the tech community beyond "Apple is trying to prevent child porn."

And who exactly will the market share go to? Do you thing the privacy-conscious folks will go to Google? Or a dumbphone? Or Purism maybe? I mean, no need to fret over my photos if I can't take photos, right?

I doubt that it was universal. Nothing ever is.

But I haven't seen any positive discussions about it, which is odd.

Case in point, I'm pretty apathetic about the whole thing

Here's an example of a confused commenter in this very thread. => https://news.ycombinator.com/item?id=28175002

There are legitimate privacy concerns, and Hacker News users are right to be upset. But that's not an excuse to pretend like this issue is broadly understood.

I’m sorry, at what point there was any confusion that Apple is going to use our phones to do scanning?

That is the where of the story. There was a inarguably confusion over the when, what, and how of the story.

It was not universally understood that this would only apply to photos sent to iCloud.

It was not universally understood that this was only looking for previously known CSAM.

It was not universally understood that they were using some sort of hash matching so photos you took yourself would not trigger the system.

I understand if you consider the where more important than the others, but it is simply a fact that there was confusion on what exactly was happening here.

I think "universally understood" is doing a lot of work to portray a much higher degree of certainty about each of those statements than is justified.

A lot of the contention wasn't about the specifics of their plan, but rather how subtle changes could vastly expand the scope of their plan.

"this would only apply to photos sent to iCloud." for now, until scope creeps.

"this was only looking for previously known CSAM." for now, until scope creeps.

"using some sort of hash matching so photos you took yourself would not trigger the system." well this one is immediately concerning even within claimed scope because there ARE going to be false positives that apple records some database. Millions of iphone users are going to have a non-zero "possible childporn" score.

They are building an engine for iphone users to self-incriminate. If they rigidly hold the scope to only what they announced and never expand, it could be argued that this is a reasonable concession to fight CSAM. However, in making the announcement, they boldly stepped past their existing hard line in privacy (local device content is private and not surveilled by apple), so it seems naive to expect that this announcement reflects the eventual scope of this self-incrimination engine for the next decade of apple updates.

> ... there ARE going to be false positives that apple records some database. Millions of iphone users are going to have a non-zero "possible childporn" score.

I don't think this is right. The apple pdf on the features says

> With the initial match threshold chosen as described above, iCloud Photos servers learn nothing about any of the user's photos unless that user's iCloud Photos account exceeded the match threshold.

This implies to me that they are using some sort of encryption to prevent iCloud from learning even how many matches there are until the threshold is met.

>A lot of the contention wasn't about the specifics of their plan, but rather how subtle changes could vastly expand the scope of their plan.

The how helps show us how changing this system is not a subtle change. It isn't like they can flip a switch and suddenly they are identifying new suspected CSAM on people's phones. That would require a new system since the current one is only hash matching.

>However, in making the announcement, they boldly stepped past their existing hard line in privacy (local device content is private and not surveilled by apple), so it seems naive to expect that this announcement reflects the eventual scope of this self-incrimination engine for the next decade of apple updates.

This is an arbitrary line that is being drawn. These are photos that are marked for sending to iCloud. Whether the scanning happens on the phone before they are sent or in the cloud after they sent is largely immaterial when it comes to the impact of the code. People are acting as if the line Apple drew was motivated by technology. That was never the deciding factor. Technology is the easy part here. That line was only a policy line and that policy has not changed. Only photos that are sent to iCloud are scanned. If you fear Apple changing that policy going forward, you should have always feared Apple changing that policy.

> It isn't like they can flip a switch and suddenly they are identifying new suspected CSAM on people's phones.

This is a strawman. Identifying *novel* CSAM is a very hard problem to do accurately - the reason they can't flip the switch is because they don't have the technical capability. All of the other things people are concerned about are things that apple does have the capability to do.

EDIT to reply since at max thread depth: *novel* image detection was never on the table, I think you missed that word

> This is an arbitrary line that is being drawn.

It seems the vast majority of people in this thread disagree that this line is arbitrary.

EDIT: whether the threshold is "verging on impossible." depends entirely on the effective false positive rate. If apple's claimed 1 in 1 trillion rate is true, it's probably not a concern. However, I find it unlikely that perceptual hashes on portions of images won't have higher false positive rates when subject matter is similar (non-CSAM legal adult porn, or images of children in swimsuits, etc). If that rises to 1 in 1 million for these types of images, that's hundreds of thousands of people being falsely accused.

>This is a strawman. Identifying novel CSAM is a very hard problem to do accurately - the reason they can't flip the switch is because they don't have the technical capability. All of the other things people are concerned about are things that apple does have the capability to do.

You just used this as an argument against this system. Why do you fear this if you don't think Apple can even accomplish this technologically?

>It seems the vast majority of people in this thread disagree that this line is arbitrary.

I would argue that people who believe that this decision crossed that line were being naïve to not have always known this was a possibility. I don't think this move brings us any closer to Apple scanning our devices for anti-government memes or whatever the fear is because what was stopping that was always more policy than technology.

Also to go back to your earlier comment, in the time since you posted it has now been revealed that this system needs to trigger 30 times before any action is taken. The odds of 30+ false positives is likely verging on impossible.

It's software. it will have bugs. lives of innocent people will be ruined.

I'm a huge Apple fan and a Mac devotee of 30+ years. I love my iPhone. If this thing becomes real, I will go shopping for another phone.

This is absolute insanity. I can't believe they even thought about launching this.

I'm completely disgusted.

> It seems the vast majority of people in this thread disagree that this line is arbitrary.

Maybe. I wonder how many people are choosing not to respond because they’ll only be voted down by the people who feel very strongly about this. I’ve curtailed and hedged many my contributions to this topic on HN because of this.

This is a common issue when social media is used to debate matters which are highly emotionally asymmetric.

Don't self sensor. You have an opinion, voice it. Downvotes be damned. You ain't gonna change anyone's mind if you don't say anything.

This in my mind is the mentality that contributes to a perpetual outrage culture. Standing up for your beliefs and trying to convince others by argument is exhausting and not always effective (see: Brandolini's law). My preference is to just leave my thoughts without the intention of trying to convince anyone, and to learn what can be learned. Sometimes it is necessary for us to disengage when constantly arguing isn't worth the effort.

Most of the time I'm quite outspoken. But it's clear nobody on HN is willing to change their mind on this issue. I'm just not interested in the emotional investment only to deal with oblique inferences that I'm stupid or don't understand their arguments.

It was not universally "understood" because it was not universally agreed upon. To wit:

* It was not universally understood that this would only apply to photos sent to iCloud.

Since the scanning doesn't happen on iCloud, this distinction is irrelevant.

"We are going to intrusively scan the subset of your photos that you care enough to back up to the cloud that we've been pushing to you for years" is pretty clear.

* It was not universally understood that this was only looking for previously known CSAM.

It was only looking for whatever is in an opaque database which, according to a third party we don't have any contract with, contains CSAM.

* It was not universally understood that they were using some sort of hash matching so photos you took yourself would not trigger the system.

Yeah right, I feel totally safe knowing that I won't be falsely reported to FBI by a "some sort of" hash matching.

Here's a hash function: f(x) = 0 for all x

It's "some sort of" hash, too.

> Since the scanning doesn't happen on iCloud, this distinction is irrelevant.

It’s really not irrelevant. A third party photo library that avoids using the PhotoKit library would not be touched by the CSAM detector. There are many of these on the App Store.

One step further and store the photos encrypted, with a custom renderer that decrypts the content on the heap, and that would take some tremendous performance-hitting detection abilities it’s extremely unlikely to ever happen.

"Look, there's a hack that allows you to avoid this at the cost of convenience that you bought the device for!"

OK.

“Don’t support this company and it’s entire product line because this one app is scanning for kiddie porn”.

Misrepresenting each other’s arguments is not how you have a discussion.

"Don't support this company and it's entire product line because the software on their entire product line is going to call a SWAT team on your location if it thinks it matched something in a database of what some other guys say is kiddie porn, but nobody can see!"

I hope that's precise enough, and sure, everyone is welcome to make their own decisions regarding this.

Precisely wrong. It’s just iCloud photos.

There's always going to be plenty of people commenting who didn't even bother to read the article at all. But by and large, from what I saw, people did understand the nuances of this and outlined how little stands in the way of expanding these policies' scope once the technology is in place

You forgot the who

- Whose fault is it that those points were not clearly communicated?

- Who wrote the perceptual hash matching code?

- Who is allowed to audit the code, the review system, and the hash database?

- Who updates this code?

- Who decides if your phone OS is updated?

- Who decides the iCloud upload defaults?

- Who decides if you are reported?

- Who asked for this feature?

- Who is going to have their dog killed by a misguided SWAT team that doesn't bother analyzing the automated report before acting on it?

I appreciate your more detailed clarification and agree with ‘where’ conclusion.

To the extent that other parts of this story was explained to us by Apple, I did try to clarify some exaggeration in other thread.

Your problem is trusting evil people at face value.

> some sort of hash matching so photos you took yourself would not trigger the system.

This is ignorance in extreme.

Nobody's confused about that.

> HN is a tech forum and I still saw plenty of people here worried about how they would get in trouble for having innocent photos of their own children on their phone.

They're confused about this. NeuralHash doesn't look for pictures of naked kids. It looks for pictures that are identical to the ones they've put in their signatures list.

The problem is that Apple claims that the signatures in their list are all pictures of sexually-abused kids, but we have no way of verifying that. Heck, they don't even have any way of verifying that. Everyone just has to take NCMEC's word for it.

Perceptual hashing can have collisions, and they will be at a higher rate than completely "identical".

The public does not know what the false positive rate is for 'average iphone user pictures'. As engineers we can be certain the false positive rate is not zero. This means that some number of iphone users are going to have non zero "possible child pornographer" scores in the apple database.

The false positive rate is crucial to understanding how concerning this should be. If the average iphone user has 1000 photos, and the false positive rate is the claimed 1 in 1 trillion, there is a 1 in a billion chance that you'll be flagged as a potential child pornographer. (~1 in the world will be falsely accused). This seems reasonable enough with the apple-internal screening step.

If the chunking and perceptual hashing functionally ends up having a much higher false positive rate for images which have similarities to the dataset (parents' pictures of kids playing shirtless, legal adult porn, etc), the false positive rate could actually be more like 1 in 1 million or worse. In which case there are potentially hundreds of thousands of people who will be falsely accused by this system.

How many matches will US judges require before they sign warrants for arrests, search and seizure of digital devices? If they are technically competent it shouldn't only be 1, but I don't trust all judges to understand probability well enough to require multiple matches.

> How many matches will US judges require before they sign warrants for arrests, search and seizure of digital devices? If they are technically competent it shouldn't only be 1, but I don't trust all judges to understand probability well enough to require multiple matches.

Even that doesn't come without issue. How long before '1' becomes the value, because, say for example the number is ten, there's also a horrendous PR spin of "Apple has a high degree of suspicion that you have CSAM on your device, but since there's only 8 images, they won't do anything about it" - "Apple allows up to X non-reported CSAM images on Apple devices" is hard to represent in any positive fashion.

Even if we put aside collisions, new system features ‘Synthetic Match Vouchers’, which is seemingly adding noise into actual CP counter.

I yet to understand what happens to people who only have those synthetic positives? Regardless of what counter threshold is, can’t those people be hoovered up by a subpoena of counter >0 ?

Oh yikes, I didn't realize they were handing out fake CP points to preserve privacy.

That's really really really user-hostile design.

You’re also wrong no? The perceptual hashing doesn’t match EXACTLY the same photos, it intentionally matches photos similar to the same photos, so a minor rotate by a few degrees or crop or whatever also matches

Why does it matter that much which CPU runs the check?

Not confusion - we will eventually remove humans from CSAM checking and it will all be AI's. The "slippery slope" is already halfway down the hill.

Or even without calling it 'AI', perceptual hashing like you typically see in applications like CSAM is pretty damn close to ML techniques. The normal thought process is "we don't want child predators to sneak through just by cropping, or sticking a water mark on the image, or some other way slightly modifying the image like the color balance. Can we come up with something that'll hash the same even with minor modifications to the image?". And you basically end up with intentionally overfit AI.

You could argue that the slippery slope started when we invented the transistor :)

True, but a small point of contention compared to the introduction of on-device scanning for illegal activities. The policies of which could be changed at a moment's notice. Other details are relatively unimportant.

This is in essence the problem with the new on-device initiative from Apple (calling it a tool is rather misleading).

If allowed to go forward, it is only a matter of time before the capability is expanded.

So it's a big no to the scanning capability, you would think that Apple had gotten the message by now.

And the other initiative is also open for abuse, by allowing the device administrator to spy on the user. Admittedly not as bad as the on-device scanning.

I don't think that was confusion either, because there were discussion and articles on how a hash collision is possible in the scenario you mention due to the way perceptual hashes work.

They have said that the system won't be triggered on a single image. You would need to have multiple photos on your phone experience this hash collision which drops the odds of false positives considerably.

EDIT: It has now come out that you need to trigger the system 30 times before Apple acts on it. I can't imagine the odds for someone to have 30 hash collisions.

So they understand that their system is very susceptible to false positives, but they are saying their clients shouldn't worry because the black box hash gets compared with a black box inauditable threshold, both of which solely determined by Apple. I don't think the reaction was due to any confusion. People understood what Apple was trying to do and realized how much it sucked from a technical perspective.

Where did "very susceptible to false positives" come from? If the odds of a collision are one in a million that is troublesome if they only need one match. If they ignore anyone that has less than 3 matches, we don't really have to worry about false positives. People who have CSAM generally don't have only 1 or 2 images.

So why 30 matches before they do something? Why not 100? Why not 5?

I assume because they know the system is so false-positive-prone that even 29 false matches is a regular occurrence not worthy of wasting an employee's time to look at.

There is a good amount of confusion throughout this very thread about what this is.

They are talking about how there is no confusion about the overreach in scanning on people's devices. You are arguing a straw man.

There's plenty of confusion, but what's telling in this story is that the people who are the least confused are among the most opposed to Apple's plans.

They are absolutely correct to be worried.