rvanderlinden

Thanks! It already works fully local!

  overtone transcribe video.mp4 --local

STT runs on whisper.cpp, and the default LLM is qwen3:8b.

For a different model, use --llm ollama:gemma3:12b (after ollama pull gemma3:12b)

If you add your name into the prompt that adds context for the LLM-check, it will from context use "Simi" and "see me" correctly. Feel free to try :)

This is the same mechanism that changes "Tom" to "Thom" in the Tears of Steel demo (turn off elevenlabs for that run because it magically knows when it's transcribing ToS ;)

It will probably even be able to correctly pick it up in a longer text purely from context, though it might still miss-spell your name because it has no way to know the actual spelling. Same would be true for a human subtitler btw :)

I've seen this a while ago, it's not really new behavior.

Only using Opus 5.5 and quality has been pretty consistent tbh.

Awesome, please give us your feedback, the current version was purely built for ourselves so we probably missed things other people need.

It's pretty simple right, a frontier LLM like Opus 5.5 has substantially better results than a cheap locally-run model.

Talking about that btw, gpt-6.1-terra scored substantially worse on fixing punctuation in our tests than gpt-6.1-sol. Didn't expect that really.

It currently only supports a handful of languages but there's no limit to what can be added really.

Only one language per video is tested currently, multi-language is a bit trickier but would probably still work, especially if it's just a few terms in another language. Haven't tested.

Netflix does this as well, they did it for a Mexican show called Ctrl-Z: https://www.youtube.com/watch?v=amyCefA5PfM

And here's an example of the Warner Bros. movie Demolition Man, where they localized Pizza Hut product placement outside of the US: https://www.youtube.com/watch?v=mn-PD4VuG8g (even the logos on the door were paintbrushed away, there's more clips to be found on YouTube btw)

Both Netflix and Disney push development of the Interoperable Master Format (IMF). I haven't seen visual localization happen outside of those two.

The John Wick movies have "expressive" closed captions, which as far as I know they've done in both English and Spanish. Other languages get forced narrative subtitles, sometimes with and sometimes without the English burnt-in captions.

And about the future: GenAI could definitely help localize parts of the video. Wouldn't surprise me if it's been used for this already. Combine that with IMF and you've got a solution that can instantly be adopted in the high-end production world.

Misheard words is something that Overtone fixes. But it can't fix all, in some cases audio mix and pronunciation are so bad that it's genuinely hard even for a human in their native language to understand. Music also messes up speech-to-text models.

Unfortunately translation of (on-screen) text wasn't built into Overtone. This is something both Netflix and Disney are very aware of, and they already create multiple visual versions of content. On both platforms you can trigger a different visual language version by changing your UI language. Disney movies are most known for this: https://www.youtube.com/watch?v=FxzL7UYyAeg

They use the IMF format to transport content with "overlays" for different languages. In those languages certain frames are swapped out for a translated version. So that an EXIT sign is seamlessly translated to a SALIDA sign in Spanish for example.

Simpler way to do this is forced narrative captions. They are basically a separate captions track to only translate stuff like the on-screen sign you mention. This is combined with regular captions. Netflix turns them on by default I believe, and on Apple TV+ it's called "auto" captions.

The core problem is that AI isn't yet able to do everything. There's a race to the bottom going on for translation companies and things won't get better until AI gets better probably. Companies like Netflix and Disney still spend good money to get high-quality captions, but smaller ones don't unfortunately. Best thing for you to do might be to message the provider so they at least know that people are unhappy.