Thanks for your fork, I know a lot of folks love it. This might not be the HN style, But I'd suggest you a freemium microSaas approach. A lot of people I know want 1-click solution for audios and you might have great potential for that. That can add some $$$$ to your bank.

Thanks. I'll consider it but I haven't deployed a model or service like that, so it'd be more of a second project in itself than a funding mechanism, probably.

I wish there was an easy way to fine tune bark, so we could truly clone our voice for bark inference.

Sadly the bark-voice-clone fork doesn't do it. The voices sound nothing like yourself.

Your gradio gui is great. But I don't understand where to copy the cloned npz files to. Even after refreshing the gradio GUI, the ClonedVoices don't appear in the Speaker or Generated Speaker dropdown.

I barely touched that, it's just from the Serp cloning github, but people kept asking so I put it in. Their clone just isn't really doing much though, it's just loading up the coarse model with the encoded wav file as a fake last generation history to the current segment, and that's basically it. But a lot of the voice is in the semantic model so just doing the coarse doesn't get you much. And even if the semantic model didn't matter as much, the coarse model has both semantic and coarse tokens as inputs and your injected coarse tokens aren't going to line up just right with what a true Bark generated pair of tokens would look like. So what you get is like a robot clone that has the most superficial similarity and lacks the depth of cadence that makes Bark awesome. That's in best case, more often get voices full of static or that don't speed the text you give them.

I'm sure somebody will train a model that actually maps an input text to the Bark semantic representation, it shouldn't be that hard, it's just been a few weeks. But the existing clone there is just so primitive I don't know how it got so popular.

> I’m sure somebody will train a model that actually maps an input text to the Bark semantic representation,

But, clearly, that’s already part of the Bark model, so in the abstract one should be able to leverage the existing model with appropriate code to do that, rather than developing a new model. This seems too simple but…isn’t that just “generate_text_semantic” from the existing source (with None as the history_prompt. since you don’t want it in the context of some pre-existing speaker)?

EDIT: Looking at the SERP voice clone, that’s what they are doing. The one thing that I’m intuitively skeptical about (and this is way out of the kind of programming I do normally, so I could be way off) is that the temp they use is kept at the level normally used for synthesis (0.7). I’d think you’d want the temp low, since you’d want generating a baseline for a new speaker to be more deterministic than generating content from an existing speaker.

Leverage sure, but you still basically need a model that does the opposite. You can't like take OpenAI Whisper, which turns speech to text, and just run the model backwards and generate audio. For example. I mean you probably could with some work, but not out of the box.

So, we’re talking about a model that takes an input audio and returns a Bark history prompt, not text -> Bark semantic prompt, right?

If so, yeah, I agree that’s tricky.

If it’s as good as you say, set up a sponsorship goal and we’ll contribute. 11 is massively expensive to use in embedded in an application so anything that’s lightweight enough to be self hosted and can produce 11 labs like output has my dollars.

Reading the comments in your code is hilarious :-)

Thank you.