I think it boils down to that it doesn't matter what they promise, if you send a videocap of all you ever do on your computer to some company on the internet, you just have to take your chances. Would you put mics and cameras in all of your rooms in your home that send data to Apple (or someone else) to analyze "for your benefit" even if they say and promise they won't do anything bad with the feeds?

At least with gmail and chat clients etc. things are somewhat put in compartments, one of the services might screw up and do something with your emails but your Messenger or WhatsApp chats are not affected by that, or vice versa. But when you bake it into the OS (laptop or phone) you're IMHO taking a much bigger risk, no matter what the intentions are.

There is nothing which Apple Intelligence can do that a hypothetically evil Apple couldn't have done before, given sufficiently treacherous code in their operating systems. Thus if you use an Apple device, you're already trusting Apple to not betray you. These new features don't increase the number of entities one must place their trust in.

Whereas with apps like Gmail and WhatsApp on an iPhone, you must trust Google and Meta in addition to Apple, not in place of Apple. It doesn't distribute who you trust, it multiplies it.

I still think it's a big difference between trusting existing OS'es and apps, which are under scrutiny by hundreds of security researchers and thousands of security nerds all the time, and willingly sending away all your data to a party who promises they will treat it well (I know it doesn't work like this in this case, but just for the sake of argument).

In essence, what you're doing is training an assistant to learn all of your details of your life and habits and the question is if that "assistant" is really secure forever. Taken to the extreme, the assistant becomes a sort of "backup" of yourself. But yeah it's an individual decision with the pro's and con's of this.

If you were already in on iCloud, that one residual distinction is moot.

Open code, inspected and used by a large number of users, hosted on hardware you physically control

Most open source code isn't inspected though.

There have been many cases recently of compromised code being in the wild for quite some time and then only known about by accident.

100% of closed code is not inspected

By you. Three comments above references "independent audits". Meaning professional cybersecurity firms

From. what I can tell, Apple doesn't actually provide the source code itself, or provides the (cryptographically verified) binaries and VMs to run it. Reverse engineering will still need to take place, it seems.

I have been involved in security audits for 110% closed code, code that's secret even within the company.

Auditing helps the company writing it, the auditors are usually experts in breaking stuff in fun ways, and it's good for business - we could slap "code security audited by XXX" on the sales pitch.

> and it's good for business - we could slap "code security audited by XXX" on the sales pitch.

You're on the precipice of discovering the problem of incentives when it comes to audits.

Audits are good, but they're inferior to source available

I think that's fair, but impractical for most users. I have a number of Home Assistant integrations with locally hosted AI models for smart home features, but I wouldn't expect my grandma to set up a server and a few VMs when she could just give her HomePod a prompt that works with AI and have no worries about the implementation. Do you feel like Apple's "independent" auditing is insufficient?

> Do you feel like Apple's "independent" auditing is insufficient?

Yeah, pretty much

Also, your grandma might not setup a VM, but it sounds like the off-device processing is essentially stateless, or at most might have a very lightweight session. It seems like the kind of thing one person could setup for their family (with the same tamper-proof signatures, plus physical security), or provide a privacy focused appliance for anyone to just plug into a wall, if they wanted to.

I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty.

I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either.

Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.

Exactly, Apple has barely any oversight or accountability for their privacy claims. Sad to see so many people taking their word at face value.

> I have no idea what code is running on a server I can't access.

That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.)

The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction.

E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.

I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that.

Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware.

It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.