News for hackers. Stuff that matters.
Re:Llama 3.1 Omni Model by twoodfin on Wednesday September 18, 2024 @07:27PM (#41586733)
I’m not clear on the virtues or potential of a model like this over a pure text model using STT/TTS to achieve similar results.

Is the idea that as these models grow in sophistication they can properly interpret (or produce) inflection, cadence, emotion that’s lost in TTS?

Re:Llama 3.1 Omni Model by fragmede on Wednesday September 18, 2024 @07:31PM (#41586762)
Really

Yeah that's the point. Without punctuation, no one can tell what inflection my "really" above should have, but even if it'd been "Really?" or "Really!", there's still room for interpretation. With a bet on voice interfaces needing a Google moment (wherein, prior to Google, search was crap) to truely become successful (by interpreting and creating inflection, cadence, emotion, as you mentioned), creating such a model makes a lot of sense.

Re:Llama 3.1 Omni Model by Reubend on Wednesday September 18, 2024 @08:55PM (#41587340)
Essentially, there's data loss from audio -> text. Sometimes that loss is unimportant, but sometimes it meaningfully improves output quality.

However, there are some other potential fringe benefits here: improving the latency of replies, improving speaker diarization, and reacting to pauses better for conversations.

Re:Llama 3.1 Omni Model by bubaumba on Wednesday September 18, 2024 @11:11PM (#41588125)
> I’m not clear on the virtues or potential of a model like this over a pure text model

you can't put pure text with keyboard on a robot. it will become a wheeled computer.

actually this is a cool thing as a companion / assistant.

Re:Llama 3.1 Omni Model by a2128 on Thursday September 19, 2024 @12:31AM (#41588516)
There's a lot of data loss and guessing with STT/TTS.

An STT model might misrecognize a word, but an audio LLM may understand the true word because of the broad context. A TTS model needs to guess the inflection and it can get it completely wrong, but an audio LLM could understand how to talk naturally and with what tone (e.g. use a higher tone if it's interjecting)

Speaking of interjection, an STT/TTS system will never interject because it relies on VAD and heuristics to guess when to start talking or when to stop, and generally the rule is to only talk after the user stopped talking. An audio LLM could learn how to conversate naturally, avoid taking up too much conversation time or even talk with a group of people.

An audio LLM could also produce music or sounds or tell you what the song is when you hum it. There's a lot of new possibility

I say "could learn" for most of this because it requires good training data, but from my understanding most of these are currently just trained with normal text datasets synthetically turned into voice with TTS, so they are effectively no better than a normal STT/TTS system; it's a good way to prove an architecture but it doesn't demonstrate the full capabilities

Re:Llama 3.1 Omni Model by langcss on Thursday September 19, 2024 @12:53AM (#41588624)
You need a lot more power. I found gpt4o struggles doing basic OCR of printed text by hallucinating alot, while tesseract engine (old skool) gets it perfect. You need the model to be powerful enough to do everything.

You can work around this by the way by sending the output through a checking stage.

So picture -> gpt4o -> out1, picture -> tesseract -> out2, out1,out2 -> llm.

Might work for sound too.

Re:Llama 3.1 Omni Model by killerstorm on Thursday September 19, 2024 @03:11AM (#41589356)
Speech is inherently easier to represent as a sequence of tokens than a high-resolution image.

Best speech to text is already NN transformer based anyway, so in theory it's only better to use a combined model

Re:Llama 3.1 Omni Model by falcor84 on Thursday September 19, 2024 @05:50AM (#41590182)
Interesting, I've actually been using gpt4o extensively for OCR and didn't encounter any significant issues - could I ask you to please give an example of an image of (otherwise legible) text that it hallucinates on?
Re:Llama 3.1 Omni Model by schrodinger on Thursday September 19, 2024 @11:50AM (#41593130)
Same, it's perfect at OCR. Generating an image with text in it however… nope!
Re:Llama 3.1 Omni Model by langcss on Friday September 20, 2024 @01:07AM (#41598970)
I'll send you an email.
Re:Llama 3.1 Omni Model by theptip on Thursday September 19, 2024 @03:57AM (#41589570)
Lots has been written on this subject, check out OpenAI’s papers on -O for example.

Latency is a big one due to batching. You can’t really interrupt the agent, which makes actual conversation more clunky. And yes, multimodal has better understanding. (I haven’t seen analysis of perception of emotions, has anyone seen analysis of this capability for GPT-O?)

Re:Llama 3.1 Omni Model by spuz on Thursday September 19, 2024 @05:04AM (#41589949)
Personally, I'm very much looking forward to using a speech model like OpenAI's advanced voice mode to learn language. It can already do things like speak quickly or slowly which traditional TTS systems can't. Also, in theory a speech model could tell me if my pronunciation is accurate. It could correct me by repeating my incorrect pronunciation and then providing the correct pronunciation. I don't actually know how capable OpenAI's advanced voice mode is in this regard because I haven't seen anyone actually test this but I'm extremely curious to try it myself. If other voice models can achieve this then it will be an incredible tool for language learning.
Re:Llama 3.1 Omni Model by paulryanrogers on Thursday September 19, 2024 @07:59AM (#41590878)
Traditional TTS can certainly be cranked up in speed. Low/no vision users often listen at 2-3x.