That is, it seems like Apple really wanted to preserve "end-to-end" encryption, but they needed to do something to address the CSAM issue lest governments come down on them hard. Thus, my guess is, at least at the beginning, they saw this as a strong win for privacy. As the project progressed, however, it's like they missed "the forest for the trees" and didn't quite realize that their plan to protect end-to-end encryption doesn't mean that much if one "end" is now forced to run tech that can be abused by governments.
This kind of thing isn't that uncommon with large companies. To me it seems very similar to the Netflix Qwikster debacle of a decade ago. Netflix could very clearly see the writing on the wall, that the DVD-by-mail business needed to morph into a streaming business. But they fumbled the implementation, and what largely came across to the public was "you're raising prices by 40%."
I give Netflix huge props for actually reversing the decision pretty quickly - I find that exceptionally rare among large corporations. Remains to be seen if Apple will do the same.
You would think apple would get ahead of this story and mention … or maybe they don’t have any E2E plans at all.
iMessages are already E2E encrypted, but you are correct, iCloud backups are decryptable with a warrant (and that was reportedly added at the FBI's request). But I agree with Ben Thompson's point in that blog post, that it's OK to not have strong, unbreakable encryption be the default, and that it's still possible to use an iPhone without iCloud and get full E2E.
But with Apple's CSAM proposal is NOT possible to have an iPhone that Apple isn't continuously scanning.
I disagree completely on this. For one, users aren't aware that using iCloud means that Apple has your decryption key and can thereby read and share all of your phone's data.
And two, opt-out is a dark pattern. Particularly if you surround it with other hurdles, like long click-wrap and confusing UI, it's no longer a fair choice, but psychological manipulation to have users concede and accept.
Third, as it's hinted, smarter criminals, the ones the FBI should actually worry about, will know to opt-out. So instead the vast majority of innocent users have their privacy violated in order to help authorities catch "dumb" criminals who could have very well been caught through countless other avenues.
disclaimer: I just read Bruce Schneier's "Click Here To Kill Everybody" after the pipeline ransomware attack, and these points come straight from it.
"Do you want to enable iCloud photos? Your private photos will be uploaded encrypted to Apple's servers, so that only you can access them. Before uploading, your photos will be locally scanned on your device for any illegal content. Apple will only be notified once a sufficient volume of illegal content is detected."
An option like this, after the lawyers get at it, will be 14 pages long.
"We can change this at any time", "except by court order", "no guarantee", "not liable", and on and on and on...
This would be a large change from the way it works today, where Apple can view your iCloud photos. They’ve not made any statements indicating that is going to change.
Everything about Apples messaging makes you believe otherwise. That seems pretty disingenuous.
Then again, 99% of consumers have very little choice that doesn’t include huddles and complicated setup. We all get the same moon goo, under a differ different brands.
Good, bad, or just the fact of life?
iCloud data is encrypted at rest (edit: except for Mail apparently). The type of encryption (service or end-to-end [E2E]) is specified here: https://support.apple.com/en-us/HT202303
It can be argued that from a user's viewpoint not having E2E encryption is tantamount to not having encryption at all, but from a technical standpoint the data is encrypted.
According to the table on the second link iCloud Photos are encrypted on the server (at rest). Am I missing something?
Apple (and thus, LEO) absolutely can look at your photos on iCloud. What you are missing is that "encrypted at rest" is essentially "not encrypted in any meaningful way".
Edit: I also meant iCloud backups in my original post and how Apple can decrypt your E2E encrypted iMessages with the key the backups contain. But I posted it last night and couldn't edit it once I caught the error. It would be amazing for other iCloud services to have E2E encryption so long as the implications of iCloud backups having your encryption keys is stated front and center when choosing to opt-in.
Unsophisticated users who have lost or forgotten their passphrase can recover their data.
It's fine if you know you want to be the sole person in the world who can get to your data and have the discipline to store master passwords, and you accept Apple support cannot help you even if you have a purchase receipt for the device. But for the average older person that's not a good default.
Apple just needs to design good UX to help these people.
Perhaps a large cardboard 'key' comes with every phone. When first setting up the phone, you scan a QR code on it to encrypt your data with the key, and then you tell the owner that of they ever lose the key to their phone, they won't be able to get in again.
People understand that - it's like losing the only key to a safe.
From time to time you require them to rescan the key to verify they haven't lost it, and if they have, let them generate a new one.
It's better if the key is generated when you already have the phone.
For even better security they can instead provide an NFC smartcard or hybrid NFC + USB (like a Yubikey) where the key is stored in hardware and can't be extracted.
I really like E2E encryption of messages with proper backups on my end. It's good to know the carrier of my messages can't scan them for advertising purpose. But for my phone if I know there is no way to easily get the data back if I have an issue that seems like a major hassle with very little upsides.
I guess some people just see dark patterns where I don’t.
Making things "opt-out", without even making it hard, dramatically increases the number of people who opt-in. It can be used for many reasons.
For example, the UK switched it’s organ donation laws from opt-in to opt-out.
Another example, my government has decided that them selling their citizen's personal information to garages, insurance companies, etc, when we buy cars, should also be opt-out.
So they freely sell the car make, acquisition date, buyer's name,… [1][2]
Unless it has changed with GDPR. I’ve never bought a car.
While we're on the topic of GDPR, that is exactly why it insists on making cookies, tracking, and the "sharing of information with partners" opt-in.
And that is without hiding what you’re doing, making it unclear or confusing, or requiring multiple actions to change it.
(Links in French, sorry)
[1]: https://mobile.interieur.gouv.fr/Repertoire-des-informations...
[2]: https://www.carte-grise.org/actus/2014/05-12-L-Etat-vends-le...
We have, yet oddly, it's still called "organ donation".
How is my will involved here?
Which is the bare minimum if you want others to be able to even consider respecting it.
The default choice being, in the absence of any information, the one that increases chances of survival of actually living humans doesn’t seem misguided to me.
——
If you want compulsory and overreaching (although not necessarily bad. Got no opinion on that aspect) rules, over here we:
- can’t disinherit a descendant. The state decides the minimum each one of your genetic or legal relatives can get from your assets
- have to financially support our adult children until an arbitrary time decided by the law
- have to support and assist our parents and grand-parents if they can’t provide for themselves
- basically can be legally compelled to financially assist any ascendant or descendant
- may end up having to pay up after a divorce, even if you had an iron-clad contract, to compensate for the loss of QoL of the spouse with a lesser income
... and if you do not know that an opt-out is required to not have your organs removed, then how can it be said that you have willingly contributed them. And if your organs are removed by default, without you actually willing that this be the case, then how can this practice be called "donation"?
So instead, shall we call it "organ retrieval", "organ harvesting" and so on? All rather ugly words. But when the words that accurately describe what you are doing appear ugly, instead of looking for words which inaccurately describe what you are doing, perhaps you should ask yourself if you should be doing what you are doing.
And I'd be interested to know of the percentage of UK adults who are aware of the opt-out. I had a search around but couldn't find any surveys, are you aware of any?
Because most often, when there’s even the sliver of a doubt on the deceased one's wishes, the corpse will keep all it’s organs to the grave / incinerator.
As currently implemented, iOS will only scan photos to be uploaded to iCloud Photos. If iCloud Photos is not enabled, then Apple isn't scanning the phone.
I think the issue is that, as Ben Thompson pointed out, the only thing preventing them from scanning other stuff now is just policy, not capability, and now that Pandora's box is open it's going to be much more difficult to resist when a government comes to them and says "we want you to scan messages for subversive content".
Now though, Apple is already saying "We'll take this database of illegal image hashes provided by the government and use it to scan your phone." It's now quite trivial for a government to say "We don't have a special database of just CSAM, and a different database of just Winnie the Pooh memes. We just have one big DB of 'illegal images', and that's what we want you to use to scan the phones."
I can guarantee that some industry bigwigs are salivating at the prospect of this tech. Imagine youtube copyright strikes but local to your device.
(Yes, I know, it sounds like I'm splitting hairs distinguishing between copyright maximalism and copyright trolling. Please, humor me.)
What copyright maximalists really want is an end to the legal protections for online platforms (DMCA 512 safe-harbor) so they can go after notorious markets directly. They already partially accomplished this in the EU. It's way more efficient from their end to have one throat to choke. Once the maximalists get what they want, platforms that want to be legitimate would be legally compelled to implement filters, ala YouTube Content ID. But those filters would apply to content that's published on the platform, not just things on your device. Remember: they're not interested in your local storage. They want to keep their movies and music off of YouTube, Vimeo, Twitter, and Facebook.
Furthermore, they largely already have the scanning regime they want. Public video sites make absolutely no sense to E2E encrypt; and most sites that deal in video already have a content fingerprinting solution. It turns out it was already easy enough to withhold content licenses to platforms that accept user uploads, unless they agreed to also start scanning those uploads.
(Side note: I genuinely think that the entire concept of safe harbors for online platforms is going to go away, just as soon as the proposals to do so stop being transparently partisan end-runs around the 1st Amendment. Pre-Internet, the idea that an entity could broadcast speech without having been considered to be the publisher of it didn't really exist. Publishers were publishers and that was that.)
That really depends. And is quite a strange, perhaps worrying, take.
Because, for many people, copyright strikes are the least of their problems.
Not that I disagree on it being an issue.
Democratic countries are much more of a danger because Apple has offices in them and cannot easily pull out of the market if they feel the laws are unconscionable.
Because it’s something to push back against governments by saying « I can’t ». But it’s a societal issue if a corporation can say « I don’t want » to a government.
It’s really not the same thing and Apple just burned their « I can’t » card and are implying they can just say « no » to governments. Which is quite an even more dystopian thing.
The statement from Apple is that they will say no. Whether that actually ever happens is something we will see, one way or another.
Do you really think they would care in the slightest about banning iPhone?
Or that Putin cares more about people not being able to buy what amounts to a luxury item, than being able to hunt down opposition supporters?
And, in all honesty, there's a lot of those.
Sounds like political suicide on either side: government interference with the country's largest company, an iconic brand, for no reason other than to hopefully spy more on your citizens.
Ohh... oops.
Any chance this is already operational there?
This is incorrect.
Apple has been saying—since 2019![0]—that they can scan for any potentially illegal content, not just images, not just CSAM, and not even strictly illegal.
That's what should be opposed. CSAM is a red herring.
[0] https://www.macobserver.com/analysis/apple-scans-uploaded-co...
At last in Apple's case, ML is used only if a minor child (less than 13 years old) who is on a Family account where the parent/guardian has opted-in to the ability to be alerted if potentially bad content is either sent or received using the Messages app.
Have been whining about big data for and ML recognition systems for years.
1) The list of CSAM hashes is that provided by the relevant US government agency, that is it.
2) The project is not targeted to roll out anywhere other than the USA.
3) The hashes are baked into the OS, there is no capability to update them other than on signed and delivered Apple OS updates.
4) Apple has exactly the same leverage with EvilGov as at any other time. They can refuse to do as asked, and risk being ejected from the country. Their stated claim is that this is what will happen. We will see.
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
For example here is a broken PSI protocol in terms of point 3. I don’t think normally in PSI this is considered broken because the server knows the value so it is part of its private set.
Server computes M_s = g . H(m) . S_s
where g is a generator of an elliptic curve, H(m) is the neural hash of the image and S_s is the server blinding secret.
The client computes M_sc = M_s . S_c where S_c is the client ephemeral secret. This M_sc value is the shared key.
The client also computes M_c = g . H(m) . S_c
and sends the M_c value to the server.
The server can now compute M_cs = M_c . S_s = M_sc since they both used the same H(m) values. This allows the server and client to share a key based on the shared image.
However, what happens if the client does it’s step using the ‘wrong’ image. If 3) is to hold it should not be possible for the server to compute the key.
Client computes:
M_sc = M_s . S_c
M_c = g. H(m’) . S_c
Now server computes: M_cs = M_c . S_s = g . H(m’) . S_c . S_s
The secret shares don’t match. But if the server knows H(m’) it can compute:
M_cs’ = M_cs . inv(H(m’)) . H(m)
and this secret share will match
Normally this client side list in PSI is just used to speed up the protocol so the server does not have to do a crypto operation for every element in its set. It is not a pre-commitment from the server.
Also, maybe the way I’m doing it here is just normally broken because it is not robust against low entropy inputs to the hash function.
I've also reversed some of apple's non-public crypto that is used in some of it's services and they have made dubious design decisions in the past they have created weird weaknesses. Without knowing exactly what they are doing I would not try and infer properties that might not exist or trust their implementation.
I think it's hilarious that one of the world' great nations would scan for exactly this.
But what people are afraid of is how little it takes when the technology is there. And how Western governments seem to morph towards that. And how some bastions of freedom are falling
Even a Jew stuck in a ghetto in Nazi Germany enjoyed significantly better privacy at home from both the state and capital than a wealthy citizen in a typical western liberal nation today.
Soviet Russia and Nazi Germany in fact prove the exact opposite of what point you think you're making. They were foremost in using the "magic tech" of their day - telecommunications and motorised vehicles.
Even then, they had several rebellions and uprisings that were able to develop thanks to lacking the degree of surveillance tech found today.
Prior to motorised vehicles and the telegraph for example, borders were almost impossible to enforce on the general population without significant leakage.
Ironically in spite of powered flight, freedom of movement is in a sense significantly less than even 200 years ago when it comes to movement between countries that do not already have diplomatic ties to one another allowing for state authorized travel.
Incidentally, this is part of why many punishments in the medieval age were so extreme - the actual ability of the state to enforce the law was so pathetically limited by today's standards that they needed a strong element of fear to attempt to control the population.
Borders in medieval Europe weren't nearly as necessary because the physical ability to travel was limited. Forget traveling from Germany to France - just going to the next town over was a life-threatening affair without assistance and wealth. Legally speaking, most peasant farmers were property of their lord's manor. But practically speaking, leaving the manor would be quite difficult on your own.
Many churches railed against motor vehicles because the extra freedom of movement they made possible also broke sexual mores - someone might use that car to engage in prostitution! You see similar arguments made today about birth control or abortion.
Prior to the Internet, American mass communication was effectively censored by the government under a series of legally odd excuses about public interest in efficiently-allocated spectrum. In other words, this was a technical limitation of radio, weaponized to make an end-run around the 1st Amendment. Getting rid of that technical limitation increased freedom. Even today, getting banned from Facebook for spouting too much right-wing nonsense isn't as censorious as, say, the FCC fining you millions of dollars for an accidental swear word.
Whether or not a technology actually winds up increasing or reducing freedom depends more on how it's distributed than on just how much of it there is. Technology doesn't care one way or the other about your freedom. However, freedom is intimately tied to another thing: equity. Systems that economically franchise their subjects, have low inequality, and keep their hierarchies in check, will see the "technology dividend" of increased freedom go to their subjects. Systems that impoverish people, have high inequality, and let their hierarchies grow without bound, will instead squander that dividend for themselves.
This is a feedback effect, too. Most technology is invented in systems with freedom and equality, and then that technology goes on to reinforce those freedoms. Unequal systems squander their subjects' ability to invent new technologies. We didn't see this with Nazi Germany, because the Allies wiped them off the map too quickly; but the Soviet Union lost their technological edge over time. The political hierarchy they had established to replace the prior capitalist one made technological innovation impractical. So, the more you use technology to restrict, censor, or oppress people, the more likely that your country falls behind economically and stagnates. The elites at the top of any hierarchical system - aside from the harshest dictatorships - absolutely do not want that happening.
Capitalism is completely compatible with hierarchies and censorship - indeed one could argue that capitalism is completely incompatible with a true flattening of hierarchies since it rests on the ability of an owner class holding a monopoly over the means of production. The majority of dictatorships around the world use capitalism as the basis of it's economy. Following the dissolving of the USSR, Russia continues to be authoritarian and arguably more so than the USSR was past the Stalin era.
Aside from that, I think it's a little premature to frame the internet's ultimate effect on the world as reducing the ability of the government to censor the population when it's only been around for less than 30 years and we are already seeing mass adoption and development of both censorship and surveillance tech that goes beyond even the wildest dreams of 20th century era dictators.
I don't really buy the argument that most technology is invented in systems with freedom and equality either. It just sounds more like something we want to believe than something borne out by data. The internet and rocket ships were the product of the military - an institution that has more to do with using force to enact the will of it's host nation on others and limiting their freedoms than preserving the freedoms of their own, especially for superpowers like the USA and the CCP, which are effectively immune to conquest by military force.
This is in fact the same for the Silicon Valley and most private industry, you only think all this tech is the product of your freedom and equality because all of the actual extreme inequality and lack of freedom is kept compartmentalised to the global south through long and convoluted supply chains. It's not really freedom if only 10% of the actual people involved in the sustaining of an economy have any semblance of it -leaving aside the observation that for even this 10% that represents the average US citizen, their actual democratic agency in the state or in their job is vanishingly low.
First, the Soviet Union didn't have a prior capitalist hierarchy; the whole reason for Leninism as such was that Russia was a feudal, agrarian society, which in Marx's theory had not progressed to the stage of capitalism and therefore could not progress to communism.
Second, food shortages and mass poverty did not end with the establishment of the Soviet Union; in fact, they became enormously worse. Even before being invaded by Germany in WWII, the Soviet system caused the Holodomor, a famine unprecedented in the history of the Ukraine.
Third, the internet has been around for 52 years, not less than 30. I've personally been using it for 29 years, and I can tell you that it already had a long history when I started using it. One of the founders of Y Combinator first became well-known as a result of breaking significant parts of the internet 33 years ago, an event which resulted in a criminal trial.
Fourth, the internet was the product of universities, although the universities were funded by ARPA. Rocket ships have a long evolution including not only militaries but also recreational fireworks, Tipu Sultan of Mysore, Tsiolkovsky, Goddard, the peaceful space agencies, H. G. Wells, and possibly Lagâri Hasan Çelebi.
Fifth, I don't think the argument is that the technology is necessarily produced by systems with freedom and equality, but that it is invented by them. This is somewhat dubious, but not as open-and-shut wrong as your misunderstanding of it. Goddard's rockets were put into mass production as the V2 in Nazi Germany using slave labor, but he invented them at WPI and Princeton. Tsiolkovsky lived in the Czar's Russia and then the USSR, and his daughter was arrested by the Czar's secret police, but he himself seems to have had considerable freedom to experiment with dirigibles and publish his research (but no slaves to build rockets for him), and indeed he was elected to the Socialist Academy.
I think we can make an excellent case that certain kinds of intellectual repression, whether grassroots or hierarchical, fall very heavily on the kinds of people who tend to invent things. William Kamkwamba's family thought he was insane, Galileo spent the last years of his life under house arrest, Newton was doing alchemical research that could have gotten him burned at the stake in Spain, Qin Shi Huang buried the Mohists alive, Giordano Bruno was in fact burned at the stake, and the repression of Lysenko's intellectual opposition was a major reason for the USSR's and PRC's economic troubles in the 01950s and 01960s.
Living in the so-called "global south" (a term which I have come to regard as useless for understanding the world system, if not actively counterproductive) I have to tell you that there's very little production of advanced technology going on here. But I live in Argentina, and the situation is different in different countries; Indonesia, Thailand, Vietnam, etc., have all done significant technical production for the world economy while in the grip of dictatorships, even though Argentina never has. But most countries don't. If tantalum cost ten times as much, we'd still have cellphones, and you probably wouldn't even be able to detect the price difference.
But bad people will always have power.
Something strikes me as odd; Why do you say “Russia” but not “Germany”? Why not say “Soviets” or something similar?
Is it because Russia is still portrayed as the bogeyman in the Anglosphere whereas the Germans are cool now?
However, the difference between having a feature enabled based on the value of a seemingly-unrelated user-controlled setting, versus having that feature enabled all the time... is basically zero. Additionally, extending this feature to encompass other kinds of content is a matter of data entry, not engineering. That's the policy angle.
When you don't yet have a capability, it might take a lot of work and commitment to develop it. But on the contrary, policy can be changed with the flip of a switch.
I could maybe understand the new implementation if Apple had announced they'd also be enabling E2E encryption of everything in iCloud, and explained it as "this is the only way we can prevent CSAM from being stored on our servers."
Why is that supposed to be their responsibility?
Given the scale of their business, there is effectively a certainty that people are transmitting CSAM via Comcast's network. That's no excuse for them to ban end to end encryption or start pushing out code to scan client devices.
When you hail a taxi, they don't search you for drugs before letting you inside.
Because they're not the police. It isn't their role to enforce the law.
They (and Google, Microsoft, Facebook, etc.) are essentially mandated reporters; if CSAM is on their servers, they're required to report it.
It's like how a doctor is a mandated reporter regarding physical abuse and other issues.
Because they're not the police. It isn't their role to enforce the law.
They're not enforcing the law; if the CSAM reaches a certain threshold, they check it out and if it's the real deal, they report to National Center for Missing and Exploited Children (NCMEC); they get law enforcement involved if necessary.
I don't think that's correct. My understanding is that if they find CSAM, they're obligated to report it (just like anyone is). I don't believe they are legally obligated to proactively look for it. (It would be a PR nightmare for them to have an unchecked CSAM problem on their services, so they do look for it and report it.)
Consider that Apple likely does have CSAM on their servers, because they apparently don't scan iCloud backups right now. I don't believe they're breaking any laws, at least until and unless they find any of it and (hypothetically) don't report it
Quite honestly, my opinion is that any service not scanning for CSAM is living on borrowed time.
Scanning on their servers is much less privacy preserving for users but don't fret, Dropbox, Facebook, Microsoft, etc. already scan stored photos.
Except for the "oops, due to an unexpected bug in our code, every image, document, and message on your device was being continuously scanned" mea culpa we will see a few months after this goes live.
After they introduced AirTags I went and disabled item detection on all my devices. Yesterday I went into settings and guess what, that feature is enabled again. On all my devices! I’m not sure when that happened, but my guess would be that latest iOS update caused that…
My point is, you will only have things working to the extent you have testing for, and test coverage is likely less robust for less popular features or where perception tells developers that “no one is going turn that mega-feature off”.
Only images that match the hashes of the database of CSAM held by the National Center for Missing and Exploited Children (NCMEC) that are uploaded to iCloud Photos are checked.
Based on their technical documents, it's not even possible for anything else to be checked; even if they could, they learn nothing from documents that aren't CSAM.
Incorrect. All files on the phone will be checked against a hash database before being uploaded to iCloud. Any time before, which means all the time, if you have iCloud enabled.
A cryptographic safety voucher is created for each photo as they're uploaded. The iPhone doesn't know wether or not anything matched. Nothing happens unless the user reaches a threshold of 30 CSAM images that have been uploaded to iCloud Photos.
1. About 3 years ago, there were empty files taking up storage on my iCloud account, and they weren't visible on the website so I couldn't delete them. All it showed was that an excessive amount of storage was taken up. Apple advisors had no idea what was going on and my entire account had to be sent to iCloud engineers to resolve the issue. They never followed up, but it took months for these random ghost files to stop taking up my iCloud storage.
2. Sometimes flipping them on and off a lot causes delays in the OS and then you have to kill the Settings app and re open it to see if they're actually enabled. Back when ATT was just being implemented, I noticed it was grayed out on my phone every time I was signed in to iCloud, but was completely operational when I was signed out. Many others had this issue and there was literally no setting to change within iCloud; it was a bug that engineering literally acknowledged to me in an email (they acknowledged that it happened for some users and fixed it in a software update).
Screwups happen in an increasingly complex OS and I just feel that there will be a day when this type of bug surfaces in addition to everything that already happens to us end users.
OK, but what if Apple silently pushes out an update (or has existing code that gets activated) that "accidentally" sets that number to zero? Or otherwise targets a cryptographic weakness that they know about because they engineered it? That wouldn't require "significant modification".
Funamentally, it's closed software and hardware. You can't and shouldn't trust it. Even if they did do some "significant modification" how are you going to notice/prove it?
I see this number very quickly getting set to '1', because the spin in the opposite direction is "What, so you're saying, people get X freebies of CSAM that they can store in iCloud that Apple will never tell anybody about?"
_That_ is a whole other PR disaster.
Communication safety in Messages is only available for accounts set up as families in iCloud. Parent/guardian accounts must opt in to turn on the feature for their family group. Parental notifications can only be enabled by parents/guardians for child accounts age 12 or younger.
https://www.apple.com/child-safety/pdf/Expanded_Protections_...
If they suddenly start to break you trust - who knows what they will push to that black box next time ? You have no way to tell or prevent that, other than leaving the whole platform altogether.
We don't actually know what is implemented. We only know what Apple's PR machine is saying about it, or do we have the source code somewhere?
Not to be nit-picky but IIRC, this isn't exactly how it went down. icloud backups have always been decryptable and Apple had announced they were planning on changing this and making them fully encrypted but when pressured by the FBI, they scraped those plans and left them the way they are.
This is mistaken. If you turn off iCloud sync, Apple isn't continuously scanning your phone.
This is not accurate. Scanning doesn’t happen without iCloud photos enabled.
There's a very good reason Signal (and previously Whatsapp) were recommended over iMessage.
Not to mention the inability to swap iMessage off for SMS which has made it a favourite exploit target for hacking firms like NSO.
They don't have the keys, so how exactly would they decrypt these messages?
You can’t turn that off!!
There’s no way to tell the iPhone, “stop with the AI recognition on my photos.”
Since Apple has decided it’s going to categorize all your photos based on AI, it was just a matter of time before they started categorizing some bad stuff.
Unless you use iCloud Backup for iMessage, in which case Apple holds the keys to decrypt them.
Sure it is—just don't use iCloud Photos. They've been quite clear about this.
Even with iCloud off, the other endpoint (the phones you're iMessaging with) will be leaking your conversations to Apple because they will still have iCloud Backup defaulted to on.
iMessage is no longer an e2e messenger, and Apple intentionally preserves this backdoor for the FBI and IC.
They access 30,000+ users' data per year without warrants.
iOS 15 beta has new recovery option by secret (which is useful only on E2EE?)
And Child Safety was released most likely because the leaks. Motive of the leaks is unknown. They might be waiting for September now.
As far as encrypted backups go, it's an open question whether they want to deal with the legal and support headaches that such a change would bring. If they continued to do nothing, Congress might force their hand by legislatively outlawing stronger encryption - they had to shit or get off the pot.
For users, if you enable this feature, but then lose your password, you are entirely screwed and Apple can't help you. Encrypting "In-transit" as a middle ground is likely good enough for most people, until researchers manage to come up with a better solution.
Exactly the same with the phone if you forget your PIN/passcode. So they already do this.
Second, my comment was meant in the context of iCloud - if you don't enable the feature as described, Apple may be able to decrypt and recover something, but if you do go all the way to the logical end, yes, you're out of options.
The amount of money that is spend on their devices is just insane.
If Apple is surprised by the blowback, it’s entirely Apple’s fault because it didn’t engage with many outside experts on the privacy front.
I recall reading either in Alex Stamos’s tweets or Matthew Green’s tweets that Apple was invited before for discussions related to handling CSAM, but it didn’t participate.
Secretiveness is helpful to keep an advantage, but I can’t believe that Apple is incapable of engaging with knowledgeable people outside with legal agreements to keep things under wraps.
This blowback is well deserved for the way Apple operates. And I’m writing this as a disappointed Apple ecosystem user.
If Apple does not have decryption keys for iCloud content, they can't decrypt it, only the user can. For Apple, it's not technically feasible, and the law supports that.
Apple has now demonstrated that it is technically feasible to overcome this challenge by doing things "pre-encryption" on the user's device.
From a user's point-of-view, Apple's privacy is worth nothing. It makes no sense for them to encrypt anything, if they leak all data before encryption. Children today, terrorism tomorrow. All your messages, photos, and audio, can be leaked.
There are thousands of people at Apple involved in the release of any single feature, from devs and testers, to managers all the way up to VPs.
The fact that this feature made it to production proofs that Apple is an organization that I can't trust with my privacy.
Their privacy team must be pulling their hairs out. They better start looking for a new job.
> From a legal stand point, they are now screwed, because now they _must_ do it.
That's not the right takeaway. They're only required to turn over data to which they have access. If they continue writing software that makes it so they don't have access to the data, then they do not need to turn it over because they can't. The fact that you can write software to store or transmit unencrypted data is irrelevant.
For content that is pirated Apple would receive a slap on the wrist, but for content that shows the exploitation of children? And that Apple is hosting on their owned servers? There is no doubt every government will throw their full legal weight behind that case.
Now they are stuck between a rock and a hard place. Do they or do they not look at iCloud Photos? Well, not in the cloud i guess, but right before you upload them. Is that any better? I don't know? Probably not. They really have no choice though. They either scan in the cloud of scan on the device. Scanning in the cloud would at least let people know where their content truly stands, but then they can't advertise end-to-end encryption and on paper it may have felt like a bigger reversal.
I think the long and short answer is. Apple cannot really legally protect your privacy when it comes to data stored on their servers. The end.
Apple should use it's gigantic legal arm to set good precedents for privacy in court.
The lawyerly phrase is “knew or should have known.” It could be argued that now that this technology is feasible, failure to apply it is complicity.
Think about GDPR. Eventually homomorphic encryption is going to be at a point where “we need your data in the clear so we can do computations with it on the server” will cross from “failure to be on the bleeding edge of privacy research” to “careless and backwards.”
And nothing stops Apple from fighting against the legislative assault. They have made themselves a champion of user privacy and have deep pockets to fight the fight, yet they just surrender at first attempt.
> surrender at first attempt
The irony
Pull all apple products from the shelves for a month and launch a billion dollar ad campaign about what the hell our government is up too.
At least try to stick to one governments. Tech companies are entering tricky waters by trying to follow laws of many nations (which conflict) simply because their customers are there
When I was young, I was not effectively spied on. My activity and digital trail were ephemeral, and hard to pin down. i wasn't the subkect of constant analysis and scrutiny. I had the privilege of being invited into the engine of an idling train to share an Engineer's drinks along with my parents and no one batted an eye.
There were pedophiles then too. guess what? We didn't sacrifice everyone's right to privacy. We didn't lock more and more rights out of the reach of minors. You didn't need a damn college degree and an insane amount of idealistic commitment to opt out of the surveillance machine. I could fill most prescriptions, even out of State.
We didn't fear the monster. They were hunted with everything we could justify mustering, but you weren't treated as one by default.
I imagine that maybe, somehow, tomorrow's youth may get to experience that. Not live in fear, but in appreciation of the fact that we too, looked into things absolutely worth being afraid of, but resisted the temptation to surrender yet more liberties for the promise of a safety that will never manifest. Just change its stripes.
Those are the best nights.
No. My elders and society just denied the monsters existed. And punished any one who talked about it.
I know they’re out there in the world, but I leave my home anyways, because living my entire life curled up in my bed in fear would be a sad and short life.
I’m more afraid that people will lose the ability to be happy because they’re so full of fear and skepticism that they don’t form meaningful relationships, relax, experience life.
I’m also more afraid of what can happen when a small group of people are allowed to dictate and control the lives of many, justifying it by pushing fear. Then it’ll be more than a small group of abusers harming a small group of people (in this case, child abusers). It becomes a different group of abusers (dictators) harming the entire population through their dystopian politics.
How many people, including children, died when a group of dictators used fear & skepticism in the 1930s to push an agenda that ultimately led to WWII and concentration camps?
We need to be extraordinarily careful that we don’t try to save a very small number of lives at the expense of many lives in the future, as a result of policies that we put in place today.
Maybe they would have been surprised where it ended up, but someone in charge there is definitely on board with a reversal on the previous privacy stance, given the doubling-down with an email signal-boosting the NCMEC calling privacy concerns the "screeching of the minority".
>... to address the CSAM issue lest governments come down on them hard.
And I think that is the springboard of why this is happening.
It's no secret that Apple has lobbyists which by turns have a "pulse" on the trajectory of Bills in the House and Senate.
What immediately came to mind to me was H.R.485 of 2021's House Session:
https://www.congress.gov/bill/117th-congress/house-bill/485/...
My (layperson) estimation is that Apple knows the rewording of it is liable pass both House/Senate in some way either in the 2022 or 2023 sessions and are attempting to get ahead of the issue. (most likely as a "rider" bill at this point)
That's my thought process on what may have precipitated Apple's move toward "CSAM"; and I welcome it in some ways (unequivocally, children should not be subjected to sexual abuse) and am wary of it in others (ML is in its infancy in too many respects, and if a false-positive arises who's to say what's liable to happen on the Law Enforcement level).
There are infinitely many concerns to be discussed on both sides of this argument, and I want both to succeed in their way. CSAM, however, feels like an attempt to throw a blanket over Apple's domains-of-future-interests while playing an Altruism Card.
One thing is certain, though: they've opened the floodgates to these discussions on an international level...
Great, let the bills pass and then they get to publicly blame congress for being "forced to legally comply" instead of jumping the gun and doing an about face on 80% of your reputation built over the last 20+ years.
There is 0 reason to do this before being legally forced to. It's not a gamble with a possible upside somewhere. It's directly shooting yourself in the foot.
The iMessage thing is actual spyware, by design, it is a way for parents to spy on their children, which is fine as long as it is a choice and the parents are in control. And that's the case here.
The iCloud thing is much more restrictive, comparing hashes of pictures before you send them to Apple servers. For some reason, they do it client side and not server side. AFAIK, iCloud photos are not E2E encrypted so I don't really know why they are doing it that way when everyone else does it server-side.
But combine the news and it sounds like Apple is using AI to scan all your pictures. Which is not true. Unless you are a child and still under your parents authority, it will only compare hashes of picture you send in the cloud, no fancy AI here, and it is kind of understandable for Apple not to want to host child porn.
I am increasingly convinced this is a backroom deal, just like the iCloud Backup e2e feature (designed and, as I understand it, partially implemented) being nixed on demand (but not legal compulsion) from the FBI.
If you get big enough, the 1A doesn't apply to you anymore.
Almost everyone was traced back to the Jan 6th insurrection, because unlike antifa they took their phones with them, so easy trace for government. Now they are in solitary confinement in prison while the world loughs at their expectations of privacy from their phones.
If they get a covid passport implemented, you won't be able to go out in public without your phone to prove you've been vaccinated. So leaving a phone behind will not be much of an option if this is to pass. Metadata and contact tracing will keep track of everyone you're near. This dystopian future is possibly just a few weeks away.
I don't really see how they can recover from this. Will they go out of business? no. Will they even lose market share? No. But there will always be that example of the time Apple got it wrong.
If I was an Apple employee I would certainly be concerned about how something like this could happen.
I don't argue with your point, but actually Reed Hastings originally created Netflix the DVD rental company so that he could turn it into a movie streaming company. He had to wait for internet to be fast enough. His interview with Reid Hoffman is pretty cool:
Not just one end, but specifically the end that their users thought they had control over.
> > Apple really wanted to preserve "end-to-end" encryption, but they needed to do something to address the CSAM issue
There we are.
They could have announced this all in one shot and saved themselves a lot of blowback.
I believe they have to understand that, which makes me think the theory of them planning to add E2E may not be true.
They would have to say something like "We know some users want to fully encrypt their cloud data. We also want to encrypt their data. But the government won't allow us to do it until we protect their interests in stopping child abuse and terrorism". That would contradict their marketing image of having their interests being aligned with their users'.
Apple are scanning the images only when they are uploaded to iCloud, because they don't want the CSAM pictures on their servers.
And you're not alone. Reddit, Facebook, Twitter etc are full of shouting angry people who haven't understood the story at all.
It's almost became a Rorschach test between those who understand technology and those who like to shout at things.
This seems like an incredible reach to me.
Occam's razor also says they aren't playing 11th dimensional chess, and that they just fucked up.
First they enable this technology to prove to the government that they can still scan for bad stuff, then they enable true E2E. I don't know if this is really their motivation but it sounds plausible to me.
EDIT: though you are right - they can please governments by making encryption useless while lying to people they still have it. Not sure that was your message though.
It seems plausible to me that Apple might believe some of their privacy conscious users would prefer this situation and hence "privacy" can be considered a plausible explanation for their actions here.
And is "pressure from the FBI" really the only reason why Apple might hold backup keys? Please sketch out your "lost device, need to restore from backup" user journey. For a real person, who has only an iPhone.
How about "enter your master passphrase to access your photos". Like literally all syncing password managers work, or things like Authy work to back up your one-time password seeds. It's not complicated.
Ashton Kutcher's foundation Thorn https://en.wikipedia.org/wiki/Thorn_(organization) does a lot of work in this space. They have hashed content databases, they have suspected and watchlisted IP addresses, etc...
I don't know why I'm being downvoted. This is an active area of cryptography research, and I've applied these types of content checks in real life. You can maintain good privacy and still help combat child pornography.
Sometimes HN takes digital cynicism a step too far.
It's actually easier to the the scanning on the backend compared to scanning of the user's phones.
My opinion, maybe entirely wrong, is that it's related to how Apple operates in China. In China, Apple does not actually operate their backends, but their BE is operated by government owned business, which is "leased" by Apple.
Maybe they don't want to enable scanning on BE for that reason, and over-engineered local device scanning, that really doesn't make much sense if you think about it twice?
Sincere or not, the "the only way we could do true X is by making X utterly useless and meaningless" thing certainly seems like bureaucracy speaking. It's a testament to "the reality distortion field" or "drinking the Koolaid" but it shows that the "fall-off" of such fields can be steep.
Consider the this new hypothetical future state:
1) photos are encrypted on iCloud
2) photos are hash scanned before being encrypted and uploaded.
If you don’t upload photos to iCloud there is no change.
In all scenarios you have to trust Apple to write software that does what they say and that they won’t write software in response to a subpoena to target you specifically.
The attack with the hash scanning is someone gets a hash for something in your photos specifically (that doesn’t match a ton of people) and through some means gets it into the database (plausible), but can’t get anything without going though this route. Specifically subpoenas can’t just get all your photos without going through this route.
My conclusion is that you are more protected in the hash scan and encrypt state than in current state.
They'll ditch privacy in a second if they can sell you enough ads, just as they ditched privacy for Chinese customers when the Chinese government asked them to.
You could spin that the other way: Apple’s foray into advertising was opportunistic, but didn’t sit with their focus on privacy. You’d need internal memos to prove it went either way.
Between that and Apple's principled stance against the FBI, I'm inclined to believe that at that time, they were making principled choices out of a real concern for user privacy.
I suspect this time around they have a real concern for dealing with the growing child exploitation/abuse problem, but failed to prioritize system design principles in their zeal to pursue positive change in the world.
I don't envy them: in their position, they have an incredible amount of pressure and the stakes are incredibly high.
May cool heads and rational, principled thinking prevail.
Apple is saying this: "we need to scan your phone because we need to look for this material, and by the way, we're only scanning things _we've been scanning for years already_"
Basiclaly, everything else aside, in the current state of things, this feature makes no sense. They are not scanning anything they weren't scanning already (literally, everything on iCloud) so why did they bother?
A bunch of people are making speculative claims that Apple is doing this to enable encryption of data in iCloud, but there's no reason to believe that's true. Apple certainly hasn't mentioned it.
So .. why?
There is nothing surprising here. Apple never mentions things unless it’s ready for release or is just about to be released. It’s just how Apple works.
I can hope, I guess...
As far as we know, Apple is not already using their decryption keys to decrypt images upload to iCloud and compare them with known images. That's according to this blog post that claims Apple make very few reports to the NCMEC:
https://www.hackerfactor.com/blog/index.php?/archives/929-On...
I just don’t think the messaging they had around this is reassuring at all. While they had all sorts of technical explanations as to how trustworthy this will all be because they are in charge, the whole thing quickly went from “this is only for iCloud photos” and only in the US to “3rd party integration can happen” and “we’re expanding it to other countries”. Which I guess is a logical next step but it is a hint at expansion.
The walled garden becomes much less tempting after all of this, especially right after a scandal like the Pegasus one.
https://www.nbcnews.com/id/wbna8161067
* edit: hahaha downvotes thanks apple employees!
Once they open the door they will not be able to close it ever.
As it stands, the system is designed so that the phone snitches you out if you have CSAM on it matching some perceptual hashes (and plenty of it), but if you do not have CSAM above threshold amounts then your phone never even phones home about it. It is in fact a privacy-preserving mechanism in the sense that Apple never sees your photos.
Is it supposed to make us feel better the scanning is happening on our own devices instead of someone else’s?
It certainly increases the “creep factor.”
No it's not, it's more like in just a day they started including a chemical preservative amongst all the organic ingredients. All the good stuff is still there, it's just pointless now for people who want to eat pure organic food.
I agree with the sentiment though.
yes, but at least they are telling the truth now...
I don't quite understand why people are so against this. What if some algorithm scans your images? If you are using any cloud service to share your images then your images are already being scanned and far more invasively. Is the the storing of hashes that concerns you? Your password(s) is already being stored as a hash and we trust that it can not be reversed back to plain text, why wouldn't we trust that these image hashes can't be reversed back into images?
I've seen some notion that this is a backdoor and governments and other Lovecraftian entities could use it to suppress freedom of speech by censoring certain images, but that would require them to first have the image in question, then creating hash of the image, and then inserting it into the Apple's database and after that it would only be flagged for human verification and again assuming you trust Apple to handle rest of your digital life (such as iMessages) why wouldn't you trust Apple to share your images? If some entity has it's hooks so deep in apple that they start to censor images why wouldn't they just stop all of your messages being sent?
Please correct me if I'm missing something obvious.
You should avoid making political or moral arguments when your technical knowledge is 10 feet below your ego. An image is orders of magnitude larger than a password, meaning that many more collisions. Also, because it's a perceptual hash, even more collisions. One does not simply reverse a perceptual hash. Keep your hat on ;-)
I am not being political or moral. Why would you think that? Is this one of those "since you don't immediately agree with me it means you are the enemy and disagree on everything" type things?
>An image is orders of magnitude larger than a password, meaning that many more collisions.
Whole point of hashing algorithms is that they produce unique hashes for different inputs. Wouldn't a larger input value (bits of the image) make it harder to have collisions?
>One does not simply reverse a perceptual hash
So what actually is the privacy concern here? There is no way to get the original image from the hash at best you can get an approximation of it.
>Keep your hat on ;-)
I guess this is suppose to be some kind of dig at me. No need to be an asshole even if you don't agree with someone.
Standard cryptographic hashes should have certain properties like https://en.wikipedia.org/wiki/Avalanche_effect
However, a perceptual hash desires the opposite. Due to the pigeonhole principle, a larger space mapping to a smaller space, involves more collisions. In fact, all hashes have infinite collisions, one just tries to design hashes that don't have many collisions on smaller length inputs. Ultimately though, there will be at least 2^n collisions for all inputs n bits long if the hash is n bit output. You can easily calculate this by looking at the excess in size between input space size and output space size.
https://en.wikipedia.org/wiki/Pigeonhole_principle
Perceptual hashes are essentially designed to collide on similar data. The fine details are lost. An ideal perceptual hash algorithm would quantize as many alterable properties of an image as possible. Contrast, brightness, edges, hue, fine details, etc. In the end, you have a bunch of splotches in a certain composition that form the low dimensional eigenbasis of the hash.
Police knocking down your door will only happen if you are found to posses multiple abuse pictures and they suspect you might be endagering people.
This just feels like people constructing imaginary boogieman out of this
The question really is, will it be used for intended purposes or will it become like roadside drug sniffing dogs? A common euphemism is "probable cause on 4 legs."
https://reason.com/2021/05/13/the-police-dog-who-cried-drugs...
If it is overly sensitive, or if Apple or the government expands the list to include much more than what it has initially been slated for, it's possible that the faux detections will be used for escalation and ultimately result in parallel constructions for crimes like drug possession.
To clarify - here's an arcane scenario. Authorities receive an alert and are given permission by Apple to look at all photos taken the same day as the photo in question. The photo in question is a false match but the rest of the photos that day are still inspected. The agents observe a photo of a bag of white powder, and use it to procure further investigations/warrants. The target is later arrested for possession once physical evidence is found.
https://en.wikipedia.org/wiki/Parallel_construction
I find the laws in this nation, particularly drug laws, to be quite obtuse. I don't think it is wise to give any additional enforcement capabilities to those that are continuing to deprive citizens of life and liberty over archaic drug laws.
I hate pedophiles and child abusers as much as anyone else should. I'd execute the abusers if we are being totally honest. It's just that this technology strikes me as a potentially Orwellian device.
This might not be true, but I've always assumed that if I am suspected of something the police can already access all my iCloud (or any other cloud storage) images with court order.
What court order?
Apple spots the hash collision, snitches on you, and then police get all your photos. Isn't that how this is supposed to work?
Of course this is going to spark concern within their own ranks. It's like working for a food company that claims to use organic, non-processed, fair-trade ingredients and in just a day deciding that you're going to switch to industrial farming sourcing and ultra-processed ingredients.
It's a complete change of narrative and there's no easy way to explain it and still defend Apple's Privacy narrative, wihout doing extreme mental gymnastics.