News for hackers. Stuff that matters.
Re:Bark: A transformer based text to audio system by dragonwriter on Monday May 15, 2023 @03:23PM (#35953281)
> I’m sure somebody will train a model that actually maps an input text to the Bark semantic representation,

But, clearly, that’s already part of the Bark model, so in the abstract one should be able to leverage the existing model with appropriate code to do that, rather than developing a new model. This seems too simple but…isn’t that just “generate_text_semantic” from the existing source (with None as the history_prompt. since you don’t want it in the context of some pre-existing speaker)?

EDIT: Looking at the SERP voice clone, that’s what they are doing. The one thing that I’m intuitively skeptical about (and this is way out of the kind of programming I do normally, so I could be way off) is that the temp they use is kept at the level normally used for synthesis (0.7). I’d think you’d want the temp low, since you’d want generating a baseline for a new speaker to be more deterministic than generating content from an existing speaker.

Re:Bark: A transformer based text to audio system by JonathanFly on Monday May 15, 2023 @04:20PM (#35953968)
Leverage sure, but you still basically need a model that does the opposite. You can't like take OpenAI Whisper, which turns speech to text, and just run the model backwards and generate audio. For example. I mean you probably could with some work, but not out of the box.
Re:Bark: A transformer based text to audio system by dragonwriter on Monday May 15, 2023 @07:58PM (#35956172)
So, we’re talking about a model that takes an input audio and returns a Bark history prompt, not text -> Bark semantic prompt, right?

If so, yeah, I agree that’s tricky.