LLMs are there for generating/hallucinating text, not for understanding text.

For natural language processing you need a different kind of neural network don't you?

It's the other way around. The model is impeccable at "understanding text." It's a gigantic mathematical spreadsheet that quantifies meaning. The model probably "understands" better than any human ever could. Running that backwards into producing new text is where it gets hand-wavy & it becomes unclear if the generative algorithms are really progressing on the same track that humans are on, or just some parallel track that diverges or even terminates early.

I thought it quantifies the probability that a certain word (their output) follows a given word sequence (their training corpus and the prompt)?

Only if you wildly oversimply to the level of being misleading.

The precise mechanism LLMs use for reaching their probability distributions is why they are able to pass most undergraduate level exams, whereas the Markov chain projects I made 15-20 years ago were not.

Even as an intermediary, word2vec had to build a space in which the concept of "gender" exists such that "man" -> "woman" ~= "king" -> "queen".

> Only if you wildly oversimply to the level of being misleading.

Maybe I'm asking for an explanation :)

Since you seem to understand the mechanism, can you do a 3 line summary please?

3 lines? That's still going to be oversimplifed to the point of being wrong, but OK.

Make a bunch of neural nets to recognise every concept, the same way you would make them to recognise numbers or letters in handwiting recognition. Glue them together with more neural nets. Put another on the end to turn concepts back into words.

For a less wrong but still introductory summary that still glosses over stuff, about 1.5 hours of 3blue1brown videos, #4-#8 in this playlist: https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_...

Thanks!

... Oh interesting. And those concepts are hand picked or generated automatically somehow?

> For a less wrong but still introductory summary that still glosses over stuff, about 1.5 hours of 3blue1brown videos

Sorry, my religion forbids me from watching talking heads. I'll have to live with your summary for now. Until I run into someone who condensed those 1.5 hours in text that takes at most 30 min to read...

> Oh interesting. And those concepts are hand picked or generated automatically somehow?

Fully automated.

> Sorry, my religion forbids me from watching talking heads.

What about professional maths communicators who created their own open sourced python library for creating video content and doesn't even show their face on most videos?

My problem is with the time wasted compared to written info, not with talking heads per se.

He doesn't waste time. No fluff.

You're unlikely to get a better time-quality trade-off on any maths topic than a 3blue1brown video.

He's the kind of presenter that others try to mimic because he's so good at what he does — you may recognise the visuals from elsewhere because of the library he created[0] in order to visualise the topics he was discussing.

[0] https://docs.manim.community/en/stable/faq/installation.html

There's also a playback speed slider in YouTube. I use it a lot.

Simplifying to that point is more of what a Markov chain is. LLMs are able to generalize a lot more than that, and it's sufficient to "understand text" on a decent level. Even a relatively small model can take, e.g. even this poorly prompted request:

  "The user has requested 'remind me to pay my bills 8 PM tomorrow'. The current date is 2025-02-24. Your available commands are 'set_reminder' (time, description), 'set_alarm' (time), 'send_email' (to, subject, content). Respond with the command and its inputs."

And the most likely response will be what the user wanted.

A Markov chain (only using the probabilities of word orders from sentences in its training set) could never output a command that wasn't stitched together from existing ones (i.e. it would always output a valid command name, but if no one had requested a reminder for a date in 2026 before it was trained, it would never output that year). No amount of documents saying "2026 is the year after 2025" would make a Markov chain understand that fact, but LLMs are able to "understand" that.

I’m confident that LLM’s will not have hallucination problems in the type of requests that I send to Siri.

I don’t ask Siri for facts (just like I don’t ask LLM’s for facts). As long as it can correctly, understand what and when I ask to be reminded about something, that would be a huge improvement for me.

That and being able to map “Bedroom Fan”/“Bedroom Fan Light” to “Bedroom Fan Lights” without having to specify aliases (and even then it hearing me wrong).

I’ve see Home Assistant working with LLMs and it can understand groupings that I never explicitly defined which is very nice. I can say “Turn off all overhead lights” and it will find all my overhead lights and turn them off. Siri/Alexa can’t handle those tasks currently.

What are you talking about? We've invented the Universal Translator already.

They are AMAZING at understanding. I think even more reliably than generating. And with context, and back and forth.

Try asking ChatGPT to remember some obscure film you can only remember very hazy details about - really random stuff - I bet it will identify it for you I a few tries.

You can also totally miss spell words, use messed up grammar and it has no problems at all

If you haven't tried OpenAI's advanced voice mode, it's a mind blowing version of exactly what things like Siri really ought to become with a little more development. If that's what you mean by LLM Siri, I totally agree.

Being able to chat casually with low latency, correct yourself, switch languages mid-sentence, incorporate context throughout a back-and-forth conversation etc. turns talking to these kinds of systems from a painful chore into something that can actually add value.

But does it do actual stuff, like adding a meeting to your calendar, call people, or set a timer or a reminder?

Siri already has LLM integration...

...that will grind your request to set email Vacation Mode through the world's worst speech-to-text, jam the text into Chat GPT, and spend the next three minutes reading you an uninterruptible 3 minute essay about violence.

Over on Android it's the opposite situation. The voice interface to Google Assistant was very reliable for simple things like reminders and appointments, and even for general knowledge questions. It was part of why I didn't switch to an iPhone. Then Gemini came along, and that core functionality got a lot worse.

> LLM Siri cannot come fast enough.

I don't think it's been demonstrated that Apple could make Siri better with an LLM.

I tried this with the new ‘apple intelligence’ that I thought could see my screen.

I had a birthday invite with clear date and time: so I asked it to add to my calendar.

It just said, “add what?” repeatedly until finally deciding it needed to send it to chatGPT to help. Which it did, then just returned the text in the image without taking any action. Then I say “can you add the event now?”

“Add what?”

So I try copying the text from the image in photos and giving that to Siri to add as an event. Surly this can work?!?

Nope.

It’s just utterly pitiful.

I think that feature has been delayed.