Neural audio codecs: how to get audio into LLMs

Re:Neural audio codecs: how to get audio into LLMs by cruffle_duffle on Tuesday October 21, 2025 @11:26AM (#45656992)

Correct me if I’m wrong but you need more than just closed captions. You need precise timing too. I’d think you’d need the text to line up exactly with the audio so when the voice makes an “A” sound the text it aligns with is “A” as well.

So while having the closed captions saves some of the work, there is probably much more needed to get everything lined up.

But I’m absolutely not an expert at all. In fact this is the first I’ve ever even though about it!

[ Parent | on HN ]

Re:Neural audio codecs: how to get audio into LLMs by vvolhejn on Tuesday October 21, 2025 @12:02PM (#45657447)

Author here. Speech-to-text is more or less solved, it's easy to automatically get captions including precise timestamps. For training Moshi, Kyutai's audio LLM, my colleagues used whisper-timestamped to transcribe 7 million hours of audio.

See Section 4.2 in the Moshi paper: https://arxiv.org/pdf/2410.00037

[ Parent | on HN ]

Re:Neural audio codecs: how to get audio into LLMs by cruffle_duffle on Tuesday October 21, 2025 @12:57PM (#45658167)

Sweet!

[ Parent | on HN ]