News for hackers. Stuff that matters.
Re:Neural audio codecs: how to get audio into LLMs by amelius on Tuesday October 21, 2025 @09:26AM (#45655505)
> Many LLMs have voice interfaces, but they usually work by transcribing your speech, generating the answer in text, and using text-to-speech to read the response out loud. That’s perfectly fine in many cases (...), but it’s a wrapper, not real speech understanding.

But I can say the same about tokenization. LLMs first convert groups of characters to tokens, then use that to generate tokens, and then convert the tokens back to characters. That's not real understanding! If LLMs are so smart, we should be able to skip the tokenization step.

Re:Neural audio codecs: how to get audio into LLMs by Workaccount2 on Tuesday October 21, 2025 @09:41AM (#45655705)
Nothing is real understanding because we have no benchmark for understanding because we don't mechanistically know what understanding is. The best we have is people "vibe knowing" a benchmark that they made up on the spot.
Re:Neural audio codecs: how to get audio into LLMs by vvolhejn on Wednesday October 22, 2025 @03:42AM (#45665961)
There's a great blog post from Sander Dieleman about exactly this - why do we need a two step pipeline, in particular for images and audio? https://sander.ai/2025/04/15/latents.html

For text, there are a few papers that train the tokenization and language model end-to-end, see: https://arxiv.org/abs/2305.07185