News for hackers. Stuff that matters.
Re:Hertz-dev, the first open-source base model for conversat... by blixt on Monday November 04, 2024 @02:24AM (#42039322)
They say Hertz is first of its kind but Moshi is another duplex audio model from earlier this year that seems to perform similarly (and it runs on a MacBook): https://github.com/kyutai-labs/moshi
Re:Hertz-dev, the first open-source base model for conversat... by a2128 on Monday November 04, 2024 @03:41AM (#42039684)
Moshi never released the base model, only two conversationally finetuned models. They also never released training code except for the codec. Though I don't see any training code for Hertz either, just 3 inference notebooks, and model code full of no_grad. No paper either to help me understand how this was trained and what the architecture is like. So I'm not too sure about researcher-friendliness unless I'm missing something.
Re:Hertz-dev, the first open-source base model for conversat... by nicholas-cc on Monday November 04, 2024 @04:10AM (#42039828)
We're working on a HuggingFace release that will help with finetuning. We'd like to do a paper, after a larger release - we're a team of 4.
Re:Hertz-dev, the first open-source base model for conversat... by netdevnet on Monday November 04, 2024 @04:25AM (#42039892)
Very impressive for just 4 people. What's the team background and how long have you been working on this?
Re:Hertz-dev, the first open-source base model for conversat... by unit149 on Monday November 04, 2024 @04:43AM (#42039985)
For a rag-tag group of transcendental audiophiles operating electronic circuitry, it ionizes and atomizes well.
Re:Hertz-dev, the first open-source base model for conversat... by programjames on Monday November 04, 2024 @01:13PM (#42044405)
I'm not part of their team, but lived with them for a couple months. They've been working on it for ~5 months, and their background is 16-20 year olds who are too smart for university.
Re:Hertz-dev, the first open-source base model for conversat... by nicholas-cc on Monday November 04, 2024 @02:30AM (#42039359)
Moshi is a good model to build chat applications on, this is designed to be more of a proper base model with all the quirkiness, naturalness, and researcher-friendliness of base modeling.
Re:Hertz-dev, the first open-source base model for conversat... by underlines on Monday November 04, 2024 @09:21AM (#42041741)
- LLaMA-Omni https://github.com/ictnlp/LLaMA-Omni a speech-language model built on Llama-3.1-8B-Instruct for simultaneous generation of text and speech

- moshi https://github.com/kyutai-labs/moshi speech-text foundation model using Mimi, a SOTA streaming neural audio codec

- Mini-Omni https://github.com/gpt-omni/mini-omni multimodal LLM based on Qwen2 offering speech input and output

- Ichigo https://github.com/homebrewltd/ichigo open research project extending a text-based LLM to have native listening ability, using an early fusion technique