They won’t really be going with on-device AI, the models require way too much RAM, but maybe they claim on-device AI “pre-processes” and removes personal information before requests are sent to Apple. Apple has a marketing team to spin this better than my description.

I think on-device models would be really useful. Imagine a conversational interface with much less latency so the conversations felt real. I wonder what kind of computing power we will need before we get there (e.g. running an LLM with lots of prompt data + on device speech recognition), maybe 5-10 years?

> Imagine a conversational interface with much less latency

With current models the latency comes from processing, not from the network — going from a high-power remote server to a low-power local phone is likely to increase latency more than it reduces it

It depends on how big the model is. They are using a little LLM to help correct text and that would be dreadful if it took a server round trip.

It is being designed for the iPhone which ships with 6GB RAM.

And the idea seems for it to be more of an LAM with fallback to OpenAI for the LLM.