Actually they plan to put an LLM in your back pocket using flash memory, not silicon: https://arxiv.org/abs/2312.11514
Re:Towards 1-bit Machine Learning Models
by declaredapple on Friday March 29, 2024 @01:49PM
(#39866942)
The flash doesn't do the computations though, that's just a method of getting it to the processor
It would be better to have eeprom or some such directly attached as memory. No loading.
Re:Towards 1-bit Machine Learning Models
by lawlessone on Friday March 29, 2024 @01:28PM
(#39866711)
> could be the one to roll their own silicon and put an LLM in our back pockets.
Wouldn't they have to do a lot of catching up with everyone else already working on it?
Re:Towards 1-bit Machine Learning Models
by declaredapple on Friday March 29, 2024 @01:52PM
(#39866974)
They've been designing their own chips a while now, including with an NPU.
Also because of their unified memory design, they actually have insane bandwidth which is incredibly useful for LLMs. IMO they may have a head-start in that respect for on-device inference of large models (e.g. 1B+ params).
I don't think people are running 1B+ models on the Neural Engine these days. The high-performance models I've seen all rely on Metal Performance Shaders, which scales with how powerful your GPU is. It's not terribly slow on iPhone, but I think some people get the wrong idea and correlate an ambient processor like the Neural Engine with LLMs.
The bigger bottleneck seems like memory, to me. iPhones have traditionally skimped on RAM moreso than even cheap and midrange Android counterparts. I can imagine running an LLM in the background on my S10 - it's a bit harder to envision iOS swapping everything smoothly on a similarly-aged iPhone.
Sure, but we're discussing 1.8-bit models that, again I'm a layman, I assume are over an order of magnitude smaller in their memory overhead.
Or we could end up getting another Google integration, with iOS being too rigid and locked-down to feasibly support third-party private AI alternatives. Fate has a cruel sense of humor, sometimes.
Are you referring to the Swift version of TensorFlow?
https://www.infoworld.com/article/3608151/swift-for-tensorfl...
I believe they're referring to these rumors: https://www.macrumors.com/2024/03/21/apple-iphone-ai-talks-g...
Apple, now leaving behind the albatros that was the Apple car, could be the one to roll their own silicon and put an LLM in our back pockets.
Goodbye Siri, hello Seriously.