Rumors suggest they’re gearing up to make iOS 18 an AI focused release. It’ll be interesting to see if they offer different capabilities for online/offline scenarios, or if their offerings are strictly offline.

Here’s one story to offer some context. There are others. https://archive.is/en3VL

> Rumors suggest they’re gearing up to make iOS 18 an AI focused release.

Don't underestimate Apple at disappointing enthusiasts like you and me. We've been hearing many awesome stories about the next thing Apple will do, only to realize their marketing team chose to keep it for future iOS/MBP/iPhone generations to keep the profits high.

Running a LLM on-device alongside other apps (i.e. without it taking up all phone resources), and it being reasonably fast may well require more powerful hardware than they ever sold.

A voice assistant that takes 3 seconds to reply and then takes half a second per word is a nice demo, but not a product Apple wants to sell.

And yes, some people will say they rather have that than nothing on their hardware, but “the Internet” would say iOS 18 is slow, eats battery life, etc, damaging Apple’s brand.

Siri is often slower than this…

I think your expectations are wrong. They sit there in silence with a few leaks here and there and some github projects, people speculate and get all excited about extrapolating those things. Then Apple deliver what works which may or may not be related to it.

What they don't do is sell you a lie a year before release then deliver shit (like every other fucking vendor).

> What they don't do is sell you a lie a year before release then deliver shit (like every other fucking vendor).

If you're referring to Google, then you're right. But OpenAI has consistently delivered what they announced pretty quickly. Same with Microsoft. To think that Apple somehow has a secret sauce that helps them surprise everyone is an illusion. They've had 3 years now to show their interest in LLMs, but they're just too conservative to "think different" anymore.

I think LLMs are to inconsistent for Apple's taste, I mean that in both senses. Their perfectionism won't risk bad output, ever, which is impossible for LLMs

Nowhere near that. They’re a totally unproven tool with a lot of bad side effects. They just don’t want to go to market with that.

I mean they do truly useful stuff already using ML just not LLMs

They're plenty capable of delivering garbage. Certain year models of the MacBook pro were inherently faulty. I've had the displeasure of having two of them bought for me at work.

All of Apple's windows software (iTunes, Safari, etc) has been, at best, a barely working port.

I'm assuming they are putting a lot more thought and care into it than the touchbar, crappy keyboards and the rest, but I'm also not holding out much hope either.

Apple can screw up, no question. But they don’t do the two-year hype cycle thing that just about everyone else does in technology (or video games).

It’s incredibly rare for Apple to publicly talk about things that won’t be selling extremely soon.

The iPhone had to be pronounced because it was going to show up on the FCC website, and obviously Apple wanted to control the message. I suspect the Vision Pro may be similar, but they also wanted developers to start getting ready so they would have software day one.

The only thing I can think of that Apple pre-announced and failed at was the Air Power mat. They said it would be coming out soon after and had to push that a couple times before finally canceling it.

Other than that small exception, if modern (post jobs return) Apple announces something is coming, it will come out and be quite close to what they say.

They don’t pull a Humane AI, Segway, Cyberpunk 2077, or No Man’s Sky.

Hello MacBook Pro 2016 !

And that Apple /// that Apple released in 1980 was also garbage…

Since they got rid of Jony, it's been great. That's all I'm saying.

Absolutely!

The hardware, that is.

With software, their Come to Jesus moment is still in the future.

(To me, Swift is sort of the Jony correlate on the software side. Doesn't fit perfectly, of course, but very similar "we are perfect who cares about evidence la la la I can't hear you" vibes and results)

Yes, they have treated macOS like a toy. It's time they made it a real OS.

I despise this perspective. It’s pretty much finished. What more crap do you want shovelled onto it?

This. If there is slightest chance model can say “poop” - they’ll can it

My expectation of Apple is that they lurk in the shadows, looking at what others do while perfecting their own thing. Then on the day of release, they'll be a decade ahead of competition.

They've done this a dozen times already.

Which dozen times has Apple released something decades ahead of the competition? I'm blanking on 4-12.

Spotify / Apple Music

Netflix + Hulu / Apple TV+

Generic Earbuds / AirPods

Meta Quest / Apple Vision Pro

(The last one being a hopeful wish)

? AppleTV and Apple Music are not decades ahead of anything. AirPods are way better than the existing Bluetooth headsets that were on the market.

Maybe they aren’t “decades ahead” but they’re examples of Apple being slow to market, launching very competitive products years after the market was established.

E.g. assuming Apple Vision launches soon, they’ll be “many years behind” Quest from the date of first launch, but most likely miles ahead as far as usability.

64-bit phones is the easy one.

iPhone, Siri

> marketing team chose to keep it for future iOS/MBP/iPhone generations to keep the profits high.

VisionPro is nice. I can see costs coming down over a period of time. Also, we've been waiting long enough for that AI car.

Apple has never promised any car. From what is known, project Titan has been cancelled years ago.

The only exeption of not delivering I can recall was AirPower. It's a product they've announced and then embarassingly weren't able to finish up to their standards (or up to what was promised), so they have cancelled it altogether.

My point was, it isn't the marketing team why Apple tanks innovative/disruptive/new products.

> We've been hearing many awesome stories about the next thing Apple will do, only to realize their marketing team chose to keep it for future iOS/MBP/iPhone generations to keep the profits high.

I believe it's more likely we've heard awesome stories about things Apple will do in the future, only to realize that the average HN commenter is incapable of understanding that such stories are contextless leaks, and that it is far more likely you are operating with incomplete information than Apple's "marketing team" is holding things back for future "iOS/MBP/iPhone generations" to keep their profits high.

I know it's more fun to vomit dumb conspiracies onto the internet, but consider changing "realize" to something which conveys equivocation, because your theory about Apple's marketing team holding back mature technology in order to benefit future devices – in addition to being predicated on leaks and rumors, and risibly inane when you consider that such action would create a significant attack vector for competitors – is as equivocal as the belief that Trump is on a secret mission to destroy a global cabal of pedophiles.

I really hope they will make siri usable. In the current state it’s only good for fixed phrases. And even then it fails time to time

I hope they get rid of it completely. People on r/locallama and others have made much better assistants using GPT which use iOS APIs to control the phone. It's ridiculous that Apple still hasn't done anything useful regarding Siri.

I'd be (pleasantly) surprised; look at how long it takes them to allow system apps to get replaced with downloadable 3rd party.

And even then, the activation keyword is likely to be whatever Apple says. Similar logic as 3rd party keyboards, don't want user input to get stuck on even merely potentially untrustworthy or buggy code.

Correct me if I’m wrong but isn’t the activation word “special” in that it has to work in very low power states? I always assumed that is why the wake words are fixed, because said words need to “fit” within a very small power budget.

Could be — I've heard rumours along those lines, but none came with evidence.

https://machinelearning.apple.com/research/hey-siri (2017)

The rumors were true.

Thanks :)

That's not due to incompetence; that's due to not seeing any reason to do it.

I was not intending to imply otherwise.

It's been in an 'AI' rewrite for a while now. Pretty sure we'll see something next year.

It's been rewritten several times, people largely don't notice because they don't try using it in different languages. And of course because they want to seem savvy so they repeat impressions from other people's posts, not realizing those posts are years old.

For me fixed phrases would do it often, I use Siri mostly when driving, but keeps saying for almost any command/query: Sorry I cant do that while you are driving. (other main usecase is setting tea/cooking timers with hands full. this alone makes it pretty useful.)

I guess this won't change, because it is probably for legal reasons, to avoid being sued by some "I just dried my cat in the microwave" style genius after making a car accident (unrelated to Siri, but trying to shift the blame).

Adding support for smaller languages would be nice actually. When its reading out of Hungarian messages loud it sounds incredibly retarded. I always have a great time listening to those and trying to guess the message. :) It would be nice if I could send a message to my significant other about being stuck in traffic in Hungarian. (the iPhone keyboard already does pretty decent voice recognition in the language)

If they do it right, it might make me switch from Android. I've never used iOS before and the only thing I'm able to use Google assistant for is setting alarms, and it can't even delete the alarm I created just now.

You're unlikely to get a better experience with Siri if she becomes equipped with a 7B or 13B LLM, unless Apple figured out something revolutionary.

Released 2 days ago by Apple, a research paper on methods to run larger llms on iPhones.

https://www.macrumors.com/2023/12/21/apple-ai-researchers-ru... https://arxiv.org/pdf/2312.11514.pdf

The paper was definitely cool but doesn't allow you to run particularly large LLMs on iPhones. It allows you to run a certain kind of LLM (sparse ReLU based LLMs) whose weights are somewhere less than 2x RAM. So, 7b Falcon works, but the competitive-with-gpt-3.5-turbo LLMs are still out of reach (and aren't ReLU based, although maybe that could change in the future). And nothing is competitive with GPT-4 right now.

Of course in the long run I think it will happen — smaller and more efficient models are getting better regularly, and Apple can also just ship their new iPhones with larger amounts of RAM. But I'd be very surprised if there was GPT-4 level intelligence running locally on an iPhone within the next couple years — that sized model is so big right now even with significant memory optimizations, and I think distilling it down to iPhone size would be very hard even if you had access to the weights (and Apple doesn't). More likely there will be small models that run locally, but that fall back to large models running on servers somewhere for complex tasks, at least for the next couple years.

Yea but it's likely to be better than the current iteration of Siri even in that state.

They can still outsource to a much larger LLMs on their servers for anything that can't be done locally like they do now.

> And nothing is competitive with GPT-4 right now.

You mean nothing available? Or you mean nothing that public knows exists? The answers to those two questions are different. There are definitely products that aren't available but the public knows exist and are upcoming that are in GPT-4's ballpark.

Easy to claim but harder to prove. Name one.

I heard rumours of these claims a few weeks ago, I assume they are talking about the same thing. Nothing concrete but from a reputable person and honestly with how well mixtral performs on the chatbot arena elo board I wouldn't be surprised if it's true.

I mean nothing that is able to be benchmarked and validated by third parties is GPT-4 quality. I know there are upcoming releases that are hyped as being equal to GPT-4, e.g. Gemini Ultra, which I am very excited to get my hands on — but regardless, Ultra is not small enough to run on phones, even using the sparse ReLU flash memory optimization. And we'll see how it benchmarks once it's released; according to some benchmarks Gemini Pro has somewhat underperformed GPT-3.5-Turbo [1], despite Google's initial claims. (Although there are criticisms of that benchmarking, and it does beat the current 1106 version of GPT-3.5-Turbo on the Chatbot Arena leaderboard [2], although it slightly underperforms the previous 0613 version.)

1: https://arxiv.org/pdf/2312.11444.pdf

2: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Siri is really quite dumb. I am confident that a 7B model would be able to provide better responses in over 90% of user queries. I can't even get Siri to reliably set a timer.

Yes, Siri is really dumb. But so is every 7B/13B model out there too.

Ask perplexity7B-online anything and then compare it to siri. https://labs.perplexity.ai/

Eh no, 7B Mistral / Deepseek would certainly almost already be able to function as a super Siri, but probably something closer to PHI-2 + the new MLX apple optimisations. Have you tried those? https://twitter.com/awnihannun/status/1735355067673526360

If trained on an iPhone API + documentation and given a little web access it would blow absolutely everything out of the water.

If they can already create -basic- Python/Swift/JS/rust apps that sets timers, save things, create lists, how's that too dumb for being a Siri replacement? They just have to give it access to an iPhone/Web Api like ChatGPT's code analysis tool.

So if you ask it "hey siri do this, this and this", it will create a script, then run it on the internal API, or fetch an article then work on that etc.

I know it's still logically "dumb" but i'm not trying to play game theoretical scenarios with my phone or do logic tests or advanced math (yet).

That sounds amazing and also the jailbreak of it via adversarial voice prompting sounds like a horrific vulnerability.

True but you could make the api restricted, having certain routes completely locked, some requiring double checks, some requiring on screen approval or face-id, throttling outside fetches, only being able to run get and not etc, no financial app control etc.

But yeah "hey siri transfer all of my funds to eric", or "hey siri group all of my photos where i'm nude and send them to jack" are new almost sci fi vectors.

Depends on if they implement some form of function calling, really. If something like a 7B Mistral fine-tune had access to search and various applications, I imagine it would perform fine and better than Siri.

Note that “using an LLM” doesn’t just mean “plugging user queries straight into an LLM”. Enhancing Siri will probably be an ensemble project.

This is part of the mismatch between comparing Alexa/Siri/Cortana to a chat based LLM right now. If you just want chat and info retrieval, today’s LLMs are way better then the conversational dialogue, search, and q&a capabilities any of those assistants have. But, if you want relatively predicable task completion for things like smart home control, timers and alarms, real time weather updates, etc. (basic but frequently used interactions) or any integration with your phone or computer directly, there’s a lot to do that isn’t solved yet in the LLM space and is more of a system integration and action choice problem that the existing assistants have been hammering away at for years.

I would hope that it would at last get you out of the hell that is “what the heck did I name that stupid light?”. Device naming is, in my opinion, the worst part of any of the voice based home assistant things.

Is it “porch led”, “porch light” or “outdoor light”? Or is “outdoor light” actually the one in the front yard? What is the one by my kids nightstand? And what routine name do I use to set the mood for watching a movie?

I would hope a properly trained llm with an awareness of my devices and their locations would allow for more verbal ambiguity when controlling things.

Maybe, seems like overkill to solve a relatively straight forward problemt that is about resolution to real entities with a generative model that has no entity grounding (and may introduce other problems). It's really just not solved because Alexa/Siri leadership doesn't actually care enough about that use case to solve it. Device entity resolution and disambiguation does not require an LLM to solve for a limited number of devices, it just requires people to prioritze that over other things, and smart home device owners are the kind of early adopter that would not be the current market focus for growth (my guess, haven't worked on Alexa in many years).

I know a half dozen different ways to improve that today without LLMs.

I would argue that “info retrieval” is also something the LLM space has yet to yet to solve to a human level of reliability, but I think your comment is right on. I see this all as part of the greater symbolic vs. stochastic (/neat v scruffy) dynamics

Why would that be?

Have you ever actually used Siri?

Yes, I've been trying it out regularly ever since it was released. Last time I've talked to it for like 30 minutes while driving in October (just to test it again). It simply doesn't work for me.

I really, really doubt it for one reason: I’m convinced Apple is still terrified of that “Amazon Alexa tells child to stick a penny in a socket” story, and will hamstring themselves in an attempt to have their agential cake and eat it too

They are right to be careful, they are held to a much higher standard than their competitors.

Pixel phones have had emergency call issues for years across multiple models but they just get a pass. Apple would be crucified for this.

Sounds like a regulator issue. Doing emergency calls is a phone's #1 job, they shouldn't be allowing them to be sold if they don't work.

yet in 15+ years I have never used any of mine for that

Good for you. It's the only thing phones are required to be able to do in the US even if they don't have a working SIM or you haven't paid the phone bill.

Well, I guess they're also not allowed to cause RF interference or randomly catch fire.

sure, but that doesn't make it their #1 job

my pyjamas are regulated to be fireproof too

Apple is all about a controlled pleasant experience, it doesn't matter if it doesn't give you shiny new things; most Apple customers don't even know those shiny new things exist, so they keep spreading the word that "Apple is so easy and simple."

The idea of having an unpredictable LLM in the ecosystem is Apple's worst nightmare. I bet they will overly restrict it to the point that it stops being a general purpose LLM and becomes a neutered obedient LLM that always acts according to Apple's rules.

Also, it doesn't help that ALL the authors of this Apple paper are chinese. It raises questions about how Apple will handle political debates with its LLM.

> Also, it doesn't help that ALL the authors of this Apple paper are chinese. It raises questions about how Apple will handle political debates with its LLM.

The CCP thinks it owns all Chinese people on Earth, but that doesn't mean you have to agree with them!

With iOS 17 they added a tiny little LLM to the predictive typing. I have the newest and greatest iPhone but I feel that I very rarely see it in action. I must assume that it’s just too slow at to keep up with my typing at the moment. Or it’s just not large enough to give very many useful suggestions.

Really? Typing in my iPhone 12 Pro has become a nightmare. I suspect it is because of predictive typing ML shit. It happens all the freaking time now. The symptom is that my whole device just froze for a few seconds while the next word is squeezed out. How do I turn it off?

Compared to Android, the iOS keyboard has always been a nightmare. However, wI feel it has been causing me fewer issues within the past month-ish. Has it been updated recently?

Thankfully, Gboard is available on iOS.

Is tiny LLM an oxymoron? I believe Apple has told us it’s a transformer language model, but not specifically a LLM.

It's like how the "New Forest" is really old now: even small LLMs are (from what I've seen which isn't exhaustive) large compared to Markov language models.

Yeah, they meant a TLM

That's print('Hello, world!')

There's no difference. An LLM is just a transformer language model that's "large".

According to this article[1] it has about 34 million parameters.

https://jackcook.com/2023/09/08/predictive-text.html

It’s probably why autocomplete got drastically worse, to the point I’m considering turning it off entirely.

Most “AI” features are so incredibly fragile they’re not worth deploying.

It seems to have reset, but I find it’s actually much better than before after some intervention/training when I first updated.

Autocomplete got worse because the new system in iOS17 didn’t retain training data from prior versions. It reset everyone back to untrained. I’ve been manually correcting specialized language I use daily (e.g. “iRacing”) on my 12 (non-pro) since iOS17 release, and now it gets it correct 99.5% of the time.

So, rather than turning it off, manually correct the incorrect completions and use the suggested words bar frequently and it will learn how you type. It’s just having to start over after tossing out several OSes worth of training that makes it feel worse.

It's a GPT2 model. It hasn't changed the autocomplete experience that much (occasionally I'll see a word completion).

I have noticed Siri now describes pictures sent to Messages.

What are you going to do with it?

The auto correct is already backed by a smallish LLM, FYI.

https://jackcook.com/2023/09/08/predictive-text.html

And it is a serious quality regression IMO. The dictionary is too small and misses/messes up a ton of basic words.

It’s a place holder for iOS 18’s expansion. It’s 0.1 of the LLM in iOS. And the other prior implementation was so ducking bad that I’m not sure how you would observe such a regression.

SLM? :)

Next year releases of macOS / iOS are rumored to have LLMs as a feature .

GPT-4 voice is so, so good. Really what you would want a voice tool to be like. I can talk to it like a normal human being, unlike issuing specific commands loudly as with Siri.

But no matter the Siri shittiness (which I agree with) an LLM can only interact with the outside world – ie run commands – that exist and have a reasonable API surface, no?

Apple has had automation for ages with Automator, Shortcuts etc but nothing that actually integrates well with day to day flow. So.. setting a timer when my hands are wet already works ok, and that’s about what I need.

I honestly wonder what type of voice interactions people want with their phones. I can see transcribing/crafting chat messages I guess? But even so, it feels like it would mess up and use iMessage instead of WhatsApp, will it narrate my memes, open links and read “subscribe for only 4.99 to read this article”, cookie consents etc etc. if everything sucks how is narrating it gonna help?

Maybe I’m old but I still don’t see the major value-add of voice interfaces, despite massively improved tech and potential.

I would be happy if it does my morning routine for me. Give me a brief summary of my emails over the night, tell me the headlines from the news outlets I follow, summarize tweets from my favorited accounts, and give me an overview of the market based on my investments and watchlists.

Yes, their hardware is positioned phenomenally with little RAM even by phone standards which is what you'd hack around with for inference on mobile architectures.

Nobody can tame LLM models yet not even Apple.

I can still get chatgpt to say the most vile things and if Apple release something on device I'll get that to be a bad, baaaad robot, too.

LLMs are not yet safe for public facing production use,imo.