vvolhejn

A lot actually, since the model has all information given to it in the four views, it doesn't have to deal with any "theory of mind" of modeling the other players or being consistent over long times. See [1], there's a video of a single-player model where a car disappears behind a ball and never reappears. Multiplayer has its own risks such as the four views desynchronizing, but overall it gives a big boost to the model.

[1] https://mira-wm.com/blog-post/#hidden-information

Václav here from the team, we're happy to answer questions :) The most surprising part to me is the auto-recovery behavior we mention at the end of the blog post, since any other model I've seen always stays diverged once it goes off the rails once. But MIRA really doesn't like to be out-of-distribution. To be completely honest we're not entirely sure why this happens.

> Is this just sort of expected for these models? Should users of this expect only truncation or can hallucinated bits happen too?

Basically, yes, sort of expected: we don't have detailed enough control to precent it fully. We can measure how much it happens and train better models, but no 100% guarantee. The bigger the model, the less this happens, but this one is tiny, so it's not the sharpest tool in the shed. Hallucinated bits can theoretically happen but I haven't observed it with this model yet.

Václav from Kyutai here. Thanks for the bug report! A workaround for now is to chunk the text into smaller parts where the model is more reliable. We already do some chunking in the Python package. There is also a more fancy way to do this chunking in a way that ensures that the stitched-together parts continue well (teacher-forcing), but we haven't implemented that yet.

Václav from Kyutai here. Yes the original naming scheme was from Les Miserables, glad you noticed! We just stuck to Alba because that's the real name of the voice actor that provided the voice sample to us (see https://huggingface.co/kyutai/tts-voices), the other ones are either from pre-existing datasets or given anonymously.

I had no idea this existed, the internet is amazing

There's a great blog post from Sander Dieleman about exactly this - why do we need a two step pipeline, in particular for images and audio? https://sander.ai/2025/04/15/latents.html

For text, there are a few papers that train the tokenization and language model end-to-end, see: https://arxiv.org/abs/2305.07185

I train for 1M steps (batch size 64, block size 2048), which is enough for the model to more-or-less converge.

It's also a tiny model for LLM standards, with 150M parameters. The goal wasn't really to reach state of the art but to show how the performance of a single language model architecture can be vastly different when you just change the tokenizer.

Author here, thanks for the kind words! I think such a physics-based codec is unlikely to happen: in general, machine learning is always moving from handcrafted domain-specific assumptions to leaving as much as possible to the model. The more assumptions you bake in, the smaller the space of sounds you can model, so the quality is capped. Basically, modern ML is just about putting the right data into transformers.

That being said, having a more constrained model can also lead to some really cool stuff. The DDSP paper learns how to control a synthesizer to mimic instruments: https://arxiv.org/abs/2001.04643

You could probably do something similar for a speech model. The result would not sound as good but you could get away with much fewer parameters, because much of the modelling work is done by the assumptions you put in.

Compare also KokoroTTS, a tiny TTS that's so tiny because it uses a handcrafted system to turn text into phonemes, and then just synthesizes from those phonemes: https://huggingface.co/spaces/hexgrad/Kokoro-TTS

merci, will fix tomorrow

I use Descript to edit videos/podcasts and it works great for this kind of thing! It transcribes your audio and then you can edit it as if you were editing text.

I had a part about this but I took it out: for compression, you could keep the embeddings unquantized and it would still compress quite well, depending on the embedding dimension and the number of quantization levels.

But categorical distributions are better for modelling. It's a little difficult to explain here without using diagrams. The intuition is that if you try to have a model predict the next embedding and not the next token, you can't model multimodal distributions - you'll end up predicting the mean of the possible continuations and not the mode, which is not what you want.

Check out Section 5.3 and Figure 6 from PixelRNN, where they discuss this phenomenon: https://arxiv.org/pdf/1601.06759

At the bottom of the blog, I link two articles that do make continuous embeddings work. One of them is the Kyutai paper Continuous Audio Language Models: https://arxiv.org/abs/2509.06926

I don't know about linear models, but this kind of hierarchical modelling is quite a common idea in speech research. For example, OpenAI's Jukebox (2020) [1], which uses a proto-neural audio codec, has three levels of encoding that get coarser and coarser. They use a language model to predict continuations in the coarsest level and then have models to upscale to the finer levels and finally back to audio.

The recent MiMo-audio bunches tokens into "patches" of four timesteps and has the model predict those. [2]

[1] https://arxiv.org/abs/2005.00341

[2] https://github.com/XiaomiMiMo/MiMo-Audio/blob/main/MiMo-Audi...

Author here. Speech-to-text is more or less solved, it's easy to automatically get captions including precise timestamps. For training Moshi, Kyutai's audio LLM, my colleagues used whisper-timestamped to transcribe 7 million hours of audio.

See Section 4.2 in the Moshi paper: https://arxiv.org/pdf/2410.00037

Author here. There are a few reasons, but the biggest one is simply the compression ratio.

The OG neural audio codec SoundStream (whose first author is Neil, now at Kyutai) can sound decent at 3kbps, whereas MP3 typically has around 128kbps, as you say. Interestingly, it was originally developed for audio compression for Google Meet, not for LLMs. Today's neural codecs have even better compression.

The more modern MP3 alternative is Opus, which can work ok at 12kbps, but it's still less efficient than neural audio codecs. However, these traditional codecs are a lot less CPU-hungry, so they have that going for them.

Author here. I think it's more of a capability issue than a safety issue. Since learning audio is still harder than learning text, audio models don't generalize as well. To fix that, audio models rely on combining information from text and audio (having a single model that consumes/produces both text and audio tokens) and the audio tokens basically end up being an integrated speech-to-text/text-to-speech. This reflects my colleagues' experience working on Moshi, and it seems to be the case for other models too, see the Conclusion section.

Part of the reason can also be synthetic data: if you fine-tune on data generated from text via a text-to-speech, the tone of the voice doesn't have any information, so the model learns to ignore it.

Yes! The fact that isochrones become circles is one of my favorite things about this. I also discuss this in the video (https://youtu.be/rC2VQ-oyDG0?t=134) I made about these.

Sorry, that must be a bug. On desktop the map is large enough to be visible in its entirety but it is supposed to pan on mobile... I'll try to fix that

Indeed, you can't do this perfectly – it's an approximation. I just use one of the two directions as the travel time because if I wanted to symmetrize I'd have to call the Google Maps API twice . Asymmetry aside, it's also not possible to perfectly embed metrics in 2D Euclidean space, or any-dimensional Euclidean space for that matter (see other comment)

Oh yes, unfortunately, you can't do this perfectly. There are some graphs that cannot be embedded in Euclidean space in any number of dimensions, e.g. a 4-cycle with distance measured by path length. It's a good-enough approximation for visualization purposes, though.

Yes! This is a less practical but more funky version of isochrone maps.

Cool! I'd love to have a look at that, but the two links in the paper seem to be broken :( http://roadlessforest.eu/map.htm and http://www.map.ox.ac.uk/accessibility_to_cities/