Re:Hertz-dev, the first open-source base model for conversat...
by blixt on Monday November 04, 2024 @02:24AM
(#42039322)
They say Hertz is first of its kind but Moshi is another duplex audio model from earlier this year that seems to perform similarly (and it runs on a MacBook):
https://github.com/kyutai-labs/moshi
[ Parent
| on HN ]
Re:Hertz-dev, the first open-source base model for conversat...
by a2128 on Monday November 04, 2024 @03:41AM
(#42039684)
Moshi never released the base model, only two conversationally finetuned models. They also never released training code except for the codec. Though I don't see any training code for Hertz either, just 3 inference notebooks, and model code full of no_grad. No paper either to help me understand how this was trained and what the architecture is like. So I'm not too sure about researcher-friendliness unless I'm missing something.
Re:Hertz-dev, the first open-source base model for conversat...
by nicholas-cc on Monday November 04, 2024 @04:10AM
(#42039828)
We're working on a HuggingFace release that will help with finetuning. We'd like to do a paper, after a larger release - we're a team of 4.
Re:Hertz-dev, the first open-source base model for conversat...
by netdevnet on Monday November 04, 2024 @04:25AM
(#42039892)
Very impressive for just 4 people. What's the team background and how long have you been working on this?
Re:Hertz-dev, the first open-source base model for conversat...
by unit149 on Monday November 04, 2024 @04:43AM
(#42039985)
For a rag-tag group of transcendental audiophiles operating electronic circuitry, it ionizes and atomizes well.
Re:Hertz-dev, the first open-source base model for conversat...
by programjames on Monday November 04, 2024 @01:13PM
(#42044405)
I'm not part of their team, but lived with them for a couple months. They've been working on it for ~5 months, and their background is 16-20 year olds who are too smart for university.
Re:Hertz-dev, the first open-source base model for conversat...
by nicholas-cc on Monday November 04, 2024 @02:30AM
(#42039359)
Moshi is a good model to build chat applications on, this is designed to be more of a proper base model with all the quirkiness, naturalness, and researcher-friendliness of base modeling.
Re:Hertz-dev, the first open-source base model for conversat...
by underlines on Monday November 04, 2024 @09:21AM
(#42041741)
- LLaMA-Omni https://github.com/ictnlp/LLaMA-Omni a speech-language model built on Llama-3.1-8B-Instruct for simultaneous generation of text and speech
- moshi https://github.com/kyutai-labs/moshi speech-text foundation model using Mimi, a SOTA streaming neural audio codec
- Mini-Omni https://github.com/gpt-omni/mini-omni multimodal LLM based on Qwen2 offering speech input and output
- Ichigo https://github.com/homebrewltd/ichigo open research project extending a text-based LLM to have native listening ability, using an early fusion technique