Furthermore how private do you think Siri is? Their privacy policy explicitly states they send transcripts of what you say to them. That cannot be disabled.
Ten minutes ago i set up a new Apple device and it not only asked me if I wanted to enable Siri, but whether I wanted to contribute audio clips to improve it. What, exactly, cannot be disabled?
"When you use Siri, your device will indicate in Siri Settings if the things you say are processed on your device and not sent to Siri servers. Otherwise, your voice inputs are sent to and processed on Siri servers. In all cases, transcripts of your interactions will be sent to Apple to process your requests."
It's pretty clear and not in dispute that your transcripts are always sent to Apple.
Nonetheless, Siri is trivial to disable altogether.
Those that won’t use those won’t use this either.
This #2, so-called "Private Cloud Compute", is not the same as iCloud. And certainly not the same as sending queries to OpenAI.
Quoting:
“With Private Cloud Compute, Apple Intelligence can flex and scale its computational capacity and draw on larger, server-based models for more complex requests. These models run on servers powered by Apple silicon, providing a foundation that allows Apple to ensure that data is never retained or exposed.“
“Independent experts can inspect the code that runs on Apple silicon servers to verify privacy, and Private Cloud Compute cryptographically ensures that iPhone, iPad, and Mac do not talk to a server unless its software has been publicly logged for inspection.”
“Apple Intelligence with Private Cloud Compute sets a new standard for privacy in AI, unlocking intelligence users can trust.”
At least with gmail and chat clients etc. things are somewhat put in compartments, one of the services might screw up and do something with your emails but your Messenger or WhatsApp chats are not affected by that, or vice versa. But when you bake it into the OS (laptop or phone) you're IMHO taking a much bigger risk, no matter what the intentions are.
Whereas with apps like Gmail and WhatsApp on an iPhone, you must trust Google and Meta in addition to Apple, not in place of Apple. It doesn't distribute who you trust, it multiplies it.
In essence, what you're doing is training an assistant to learn all of your details of your life and habits and the question is if that "assistant" is really secure forever. Taken to the extreme, the assistant becomes a sort of "backup" of yourself. But yeah it's an individual decision with the pro's and con's of this.
There have been many cases recently of compromised code being in the wild for quite some time and then only known about by accident.
Auditing helps the company writing it, the auditors are usually experts in breaking stuff in fun ways, and it's good for business - we could slap "code security audited by XXX" on the sales pitch.
You're on the precipice of discovering the problem of incentives when it comes to audits.
Audits are good, but they're inferior to source available
Yeah, pretty much
Also, your grandma might not setup a VM, but it sounds like the off-device processing is essentially stateless, or at most might have a very lightweight session. It seems like the kind of thing one person could setup for their family (with the same tamper-proof signatures, plus physical security), or provide a privacy focused appliance for anyone to just plug into a wall, if they wanted to.
I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either.
Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.)
The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction.
E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware.
It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
Signal has the added benefit that it doesn't need to read what's in the messages you send. It needs some very basic routing information and the rest can be encrypted end to end. With AI stuff, the contents need to be decrypted in the cloud, so the end-to-end protections don't apply.
Apple's thrown stones come back to hunt their glass ceiling.
Secure enclave stuff can be used to build a trust relationship if it's designed well, but Apple is the party hosting the service and the one burning the private keys into the chip.
iOS won't send any data to a PCC that isn't running a firmware that's been made public in their transparency logs and compute nodes have no way to be debugged in a way that exposes user data[1]
And at the end of the day, this is going to give the warrant holder a handful of requests from a specific user? Why wouldn't they use that same warrant to get onto the target's device directly and get that same data plus a ton more?
0: https://help.apple.com/pdf/security/en_US/apple-platform-sec... 1: https://security.apple.com/blog/private-cloud-compute/
Warrants to hack devices are a lot less common and generally harder to obtain. That's why police will send Google warrants for "give us info on every device who has been in a radius of x between y and z time".
I'm sure Apple did their very best to protect their users, but I don't think their very best is good enough to warrant this kind of trust. A "secure cloud" solution will also tempt future projects to use the cloud over local processing more, as cloud processing is now readily available. Apple's local processing is a major advantage over the competition but I doubt that'll stay that way if their cloud solution remains this integrated.
Your example indicates a situation where law enforcement does not know which device belongs to their suspect, if they even have one. That's a very different scenario from a targeted "tell us the requests belonging to this individual".
Warrants to search a device are extremely common place, otherwise the likes of Grayshift and Cellebrite would not be around.
From a threat modeling perspective compromising PCC is high risk (Apple's not just going to comply and the fight will be very public, see the FBI San Bernardino fight) , high effort (Long protracted court case), low reward (I only see requests that are going to get shipped off to the cloud). If I were law enforcement I'd explore every other avenue available to me before I go down that particular rabbit hole which is exactly what this design is intended to achieve.
Once the data is out of your possession it's out of your control.
Drow "nation state is after me" from the threat model and you'll be a lot happier.
The history of tech is the history of falling costs with mass production. Expensive TLA surveillance tech for nation states can become broadly accessible, e.g. through-wall WiFi radar sensing sold to millions via IEEE 802.11bf WiFi 7 Sensing in NPU/AI PCs [1], or USB implant cables [2] with a few zeros lopped off the TLA price.
Instead of adversary motives, threat models can be based on adversary costs.
As adversary costs fall, threat models need to evolve.
[1] https://www.technologyreview.com/2024/02/27/1088154/wifi-sen...
- TLA agency deploys scarce zero days or field ops because you're particularly interesting, vs..
- TLA agency has everything about you in a dragnet, and low level cop in small town searches your data for a laugh because they know you, and leaks it back to your social circle or uses it for commercial advantage
Actually, once your e2e key that encrypts your data is out of your possession, it's out of your control.
Over the past decade it's become commercially feasible to be NSL-proof.
But in summary 1. The servers run on Apple Silicon hardware which have fancier security features 2. Software is open source 3. iOS verifies that the server is actually running that open source software before talking to it 4. This is insane privacy for AI
The security features are meant to prevent the server operator (Apple) from being able to access data that's being processed in their farm. The idea is that with that + E2E encryption, it should be way closer to on-device processing in terms of privacy and security
Here's also a great summary from Matthew Green: https://x.com/matthew_d_green/status/1800291897245835616
Apples motives are different, selling premium hardware and MORE premium hardware, they wouldn't dare fuck that up, their nestegg is hardware and slowly more services tied to said hardware ecosystem (icloud subs, tv subs etc). Hence the privacy makes sense to pull people into the ecosystem.
Google... everything google does even phones, is for more data gathering for their advertising revenue.
Apple has taken a markedly different approach, and has done so for years - E2E encryption, hashing and segmenting routes on maps, Secure Enclave, etc.
While I think it’s perfectly reasonable to “trust no one”, and I fully agree that there may be things we don’t know, I don’t think there it’s reasonable to put Apple on the same (exceedingly low) level as Google.
Apple's business model is to entice people into a walled garden ecosystem where they buy lots of expensive hardware sold on high margins. They don't need user data to make this work, which is why they can more comfortably push features like end-to-end and no-knowledge encryption.
I think also a bunch of people will trust Apple’s server more (but not completely) than other third parties.