Useful LLMs are probably still a tad to large to run locally on device. Loading and unloading the model takes several seconds, so ideally the model would just sit in RAM at all times.

In my experience playing with local LLaMA variants, ChatGPT level responses are really only possible with 13B parameter models and up. With 4bit quantization that's 6.5GB of RAM usage just for the model weights. You could totally run that on a phone today, but you'll be getting 2 tokens/s and using the majority of the phone's RAM.

A responsive, local LLM on a phone is going to happen someday soon, but we need some serious hardware improvements first. I'd speculate you'll need at least 24GB of shared memory, and considerably more dedicated tensor multiplication cores.

Alternatively, some sort of GPT architecture change that reduces the memory overhead could also drive local LLMs. Who knows, maybe Apple has a trick or two up their sleeves.

I have my doubts about them opening up their own LLMs. But it wouldn’t surprise me to see them talking about what their m-chips can run onboard.

Although, it would be pretty cool to run Siri locally. And also if she didn’t suck

Some of Siri runs locally now, much better for setting timers than it used to be.

But still way to much of

> Hey Siri, set a timer for 5 minutes

> I set your "Timer" alarm for 5:00 AM

And this...

> Hey Siri, set a reminder for next Tuesday at 11am; "Remember John arriving tomorrow"

> Ok, I set your reminder for tomorrow for "Remember John arriving"

...

> Hey Siri, change it to next Tuesday at 11am

> Ok, your reminder for tomorrow is now called "Next Tuesday at 11am"