News for hackers. Stuff that matters.
Re:VLC tops 6B downloads, previews AI-generated subtitles by jl6 on Thursday January 09, 2025 @07:27AM (#42644595)
There's an art to subtitling that goes beyond mere speech-to-text processing. Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read. Sometimes you need to name a voice as unknown, to avoid spoilers. Sometimes the positioning on the screen matters. I hope the model can be made to understand all this.
Re:VLC tops 6B downloads, previews AI-generated subtitles by diggan on Thursday January 09, 2025 @07:37AM (#42644686)
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read

Please no. Some subtitle companies do think like this, and it's really weird, like when they try to "convert" cultural jokes, and then add in a bunch of more assumptions regarding what cultures you're aware of depending on the subtitle language, making it even harder to understand...

Just because I want my subtitles in English, doesn't mean I want all typical discussed Spanish food names to be replaced by "kind of the same" British names, yet something like that is something I've come across before. Horrible.

Re:VLC tops 6B downloads, previews AI-generated subtitles by lifthrasiir on Thursday January 09, 2025 @08:12AM (#42645001)
More precisely speaking, there are two kinds of subtly different subtitles with different audiences: those with auditory imparements and those with less understanding of given language. The former will benefit from paraphrasing while the latter will be actively disadvantaged due to the mismatch.
Re:VLC tops 6B downloads, previews AI-generated subtitles by flippyhead on Thursday January 09, 2025 @07:42AM (#42644730)
I totally get this. When I'm watching videos for the purpose of learning a language, I want all the actual words in the subtitles. But if I'm watching just ot enjoy, say in a language I don't care to learn, I don't mind someone creatively changing the dialog to how it probably would have been written in English. This happens with translations of novels all the time. People even seek out specific translators who they feel are especially talented at this kind of thing.
Re:VLC tops 6B downloads, previews AI-generated subtitles by close04 on Thursday January 09, 2025 @08:16AM (#42645061)
> Spanish food names to be replaced by "kind of the same" British names

But for example you'd want the English "so so" to be translated in Spanish as "más o menos" instead of doubling down on whatever literal translation for "so" they choose.

Plenty of dialog needs language specific context, you have expressions that don't lend themselves to literal translation, or the translation in that language is so long and cumbersome that paraphrasing is an improvement.

Like with anything else, the secret is using it sparingly, only when it adds value.

Re:VLC tops 6B downloads, previews AI-generated subtitles by diggan on Thursday January 09, 2025 @08:29AM (#42645188)
> But for example you'd want the English "so so" to be translated in Spanish as "más o menos" instead of doubling down on whatever literal translation for "so" they choose.

Agree, but I don't think those are "cultural translations" but more like "idioms translations", which mostly makes sense to do.

What I originally wrote about are things like Family Guy original voice-overs + subtitles making a joke about some typical father figure in Hollywood for example. Then the Latin Spanish subtitles will have translated that joke but replaced the specific actor with some typical father figure from the Mexican movie industry, while the Castilian subtitles would have replaced it with someone from the Spanish movie industry.

Re:VLC tops 6B downloads, previews AI-generated subtitles by close04 on Thursday January 09, 2025 @09:42AM (#42646083)
I think I've encountered cases like this watching Netflix movies originally in my native language but subtitled in English. But in every case I can remember the substitution made perfect sense.

Without adapting the translation the natives will immediately understand the reference while you, the non-native with no sense of who they're talking about are left wondering what's the real message. Calling someone a "Mother Teresa" might miss the mark somewhere in China. Same if an Italian movie made references to food like Casu Marzo and the average American would probably miss a lot of the context.

Just recently I saw this in a series where some workers in the oil extraction industry were staring at a pan of paella asking what it is and calling it jambalaya. Paella is world famous, how about khash?

That's why I said I understand the usage but it should only be done if it really helps the comprehension, not just the principle of gratuitously adapting to one language.

Re:VLC tops 6B downloads, previews AI-generated subtitles by diggan on Thursday January 09, 2025 @09:52AM (#42646192)
> Without adapting the translation the natives will immediately understand the reference while you, the non-native with no sense of who they're talking about are left wondering what's the real message. Calling someone a "Mother Teresa" might miss the mark somewhere in China. Same if an Italian movie made references to food like Casu Marzo and the average American would probably miss a lot of the context.

Right, but isn't that why the American is watching this Italian movie anyways, to get a wider understanding of Italian culture? I don't watch foreign movies with the expectation that they're adapted to my local culture, then there wouldn't be much point in watching it.

Re:VLC tops 6B downloads, previews AI-generated subtitles by close04 on Thursday January 09, 2025 @10:38AM (#42646679)
> isn't that why the American is watching this Italian movie anyways

I don't know. I always though people would prefer to hear the original voice of the actor, since speech is a big part of the acting. But movies are dubbed in a lot of countries. And most people from those countries I spoke to said that they find the idea of subtitles very odd because it's distracting them from the movie.

I am sure many if not most people simply want to understand what's the message behind the conversation on the screen first and foremost. Learning something new only works if while you try to keep up with the dialogue, you also keep track of all the expressions you heard for the first time, to look them up later.

This is why I think it's the translator's job to balance translation and adaptation. Directly translate the original where context helps you understand the meaning so you get the "original" experience, and adapt where leaving just the 1:1 translation will make you lose the thread or miss some details.

Re:VLC tops 6B downloads, previews AI-generated subtitles by techjamie on Thursday January 09, 2025 @08:45AM (#42645386)
That's a good example of translation where there's only really so many ways to do it. A bad example like people are talking about is the original 4Kids Pokemon where every time someone brought out an Onigiri (rice ball), they would call them jelly donuts.
Re:VLC tops 6B downloads, previews AI-generated subtitles by scarface_74 on Thursday January 09, 2025 @07:43AM (#42644733)
I know a little Spanish and even I get annoyed when the English subtitles don’t match what they said in Spanish. Of course I expect grammatically correct Spanish to be translated into grammatically correct English.
Re:VLC tops 6B downloads, previews AI-generated subtitles by xattt on Thursday January 09, 2025 @08:20AM (#42645103)
There was an eminent Russian voiceover artist (goblin?) that translated pirated Western movies with his own interpretation.

His translations were nowhere near what the movie was about, but they were hilarious and fit the plot perfectly.

Re:VLC tops 6B downloads, previews AI-generated subtitles by naoru on Thursday January 09, 2025 @08:55AM (#42645511)
Not all of his translations were nowhere near the original. For example, his translation of Guy Ritchie's Snatch was excellent (in my opinion of course) and is still quoted to this day. I'd say it's the only one that absolutely nails it and then some.

On the other hand, his Lord of The Rings was an "alternative" dub as you described. Didn't watch that one though.

Re:VLC tops 6B downloads, previews AI-generated subtitles by bdndndndbve on Thursday January 09, 2025 @08:50AM (#42645440)
It depends on the context! Trying to Americanize Godzilla, for instance, has largely failed because Godzilla is an allegory for the unique horror of nuclear bombing which Japan experienced. Making him just a lizard that walks through New York is kind of stupid.

Jokes are an example of something translators can do really well - things like puns don't work 1:1 across languages. A good translator will find a corresponding, appropriate line of dialogue and basically keep the intent without literally translating the words.

Food is kind of silly because it's tied to place - if a setting is clearly Spanish, or a character is Spanish, why wouldn't they talk about Spanish food? Their nationality ostensibly informs something about their character (like Godzilla) and can't just be fine/replaced.

Re:VLC tops 6B downloads, previews AI-generated subtitles by diggan on Thursday January 09, 2025 @09:44AM (#42646101)
> Jokes are an example of something translators can do really well - things like puns don't work 1:1 across languages. A good translator will find a corresponding, appropriate line of dialogue and basically keep the intent without literally translating the words.

Again, those aren't "cultural translations" but "idioms translations", which I do agree should be translated into something understandable in the language, otherwise you wouldn't understand it.

What I was aiming at in my original comment was examples like these:

> Family Guy original voice-overs + subtitles making a joke about some typical father figure in Hollywood for example. Then the Latin Spanish subtitles will have translated that joke but replaced the specific actor with some typical father figure from the Mexican movie industry, while the Castilian subtitles would have replaced it with someone from the Spanish movie industry.

Re:VLC tops 6B downloads, previews AI-generated subtitles by eviks on Thursday January 09, 2025 @07:36AM (#42644679)
> better to paraphrase dialog to reduce the amount of text that needs to be read.

That's just bad destructive art, especially for a foreign language that you partially know.

> Sometimes you need to name a voice as unknown, to avoid spoilers.

Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for (also reduces the amount of text)

> Sometimes the positioning on the screen matters.

This is rather valuable art indeed! Though unlikely fit to be modelled well

Re:VLC tops 6B downloads, previews AI-generated subtitles by lxgr on Thursday January 09, 2025 @07:43AM (#42644737)
> Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for

That’s tricky when one or more speakers aren’t visible.

Re:VLC tops 6B downloads, previews AI-generated subtitles by eviks on Thursday January 09, 2025 @07:57AM (#42644859)
They'd also have to start speaking at the same time and have similar voices to make it tricky
Re:VLC tops 6B downloads, previews AI-generated subtitles by lxgr on Thursday January 09, 2025 @07:59AM (#42644881)
You’re assuming that the subtitle user can hear the voice track here.
Re:VLC tops 6B downloads, previews AI-generated subtitles by eviks on Thursday January 09, 2025 @08:02AM (#42644905)
Yes I'm, the other option is the "hearing impaired" variant, let that one have as the extra info
Re:VLC tops 6B downloads, previews AI-generated subtitles by lxgr on Thursday January 09, 2025 @09:27AM (#42645885)
I've first hand encountered several situations with subtitles where it would have been ambiguous who was speaking without speaker annotations, despite the voices being distinctive, me being able to hear them clearly etc. Just think of a rapid exchange with neither speaker on-screen for more than two or three sentences and replies.

You can probably get 99% there without that for a lot of content, but I'd challenge the notion that this is somehow only important for hearing impaired viewers (or people just watching without clearly audible sound for other reasons).

Re:VLC tops 6B downloads, previews AI-generated subtitles by soco on Thursday January 09, 2025 @09:55AM (#42646231)
I guess you can have that in a real life situation as well, where you don't have subtitles at all (hello AR) and still can handle. Do you want your subtitles full of metadata the whole movie for every movie and every day, for those several situations when the image director made a mess? You can always play that confusing scene again.
Re:VLC tops 6B downloads, previews AI-generated subtitles by lxgr on Thursday January 09, 2025 @11:54AM (#42647489)
I definitely prefer subtitles to be as helpful as possible, yes. That includes having situationally appropriate metadata (which is different from "all the metadata, all the time").

I don't think that's an unrealistic goal to have for AIs; they're already extremely good at semantic scene description after all. By looking at the image in addition to just the audio track they probably also get a lot more metadata, which a refined world model will eventually be able to use just like a human subtitle editor can today.

Re:VLC tops 6B downloads, previews AI-generated subtitles by soco on Thursday January 09, 2025 @12:25PM (#42647799)
So you mean, the AI should figure out when a problematic scene is coming, and only then add labels and whatnot? Not impossible, just somebody must teach them, same with subtitles positioning.
Re:VLC tops 6B downloads, previews AI-generated subtitles by soco on Thursday January 09, 2025 @09:51AM (#42646184)
Like other commenters pointed out, we are talking two different types of subtitles - those for hearing impaired have very different requirements. I'm not sure which one is VLC gonna cover. Best would be both, just don't mix them up please.
Re:VLC tops 6B downloads, previews AI-generated subtitles by MindSpunk on Thursday January 09, 2025 @08:31AM (#42645211)
Subtitles aren't just for foreign viewers though, they're also for native speakers who are now hearing impaired.
Re:VLC tops 6B downloads, previews AI-generated subtitles by eviks on Thursday January 09, 2025 @09:28AM (#42645907)
Sure, that's what HI version is for
Re:VLC tops 6B downloads, previews AI-generated subtitles by llm_nerd on Thursday January 09, 2025 @07:51AM (#42644800)
There is the art of subtitling, and then there is the technical reality that sometimes you have some content with no subtitles and just want a solution now, but the content didn't come with an SRT and OpenSubtitles has no match.

They're using Whisper for speech to text, and some other small model for basic translation where necessary. It will not do speaker identification (diarization), and certainly isn't going to probe into narrative plot points to figure out if naming a character is a reveal. It isn't going to place text on the screen according to the speaker's frame place, nor for least intrusion. It's just going to have a fixed area where a best effort at speech to text is performed, as a last resort where the alternative is nothing.

Obviously it would be preferred to have carefully crafted subtitles from the content creator, translating if the desired language isn't available but still using all the cues and positions. Secondly to have some carefully crafted community subtitles from opensubtitles or the like, maybe where someone used "AI" and then hand positioned/corrected/updated. Failing all that, you fall to this.

Re:VLC tops 6B downloads, previews AI-generated subtitles by Aurornis on Thursday January 09, 2025 @07:42AM (#42644727)
AI subtitles are just text representation of the sound track.

There is no need for artistic interpretation, substituting words, or hiding information. If it’s in the audio, there’s no reason to keep it out of the subtitle.

An AI subtitle generator that takes artistic license with the conversion is not what anyone wants.

Re:VLC tops 6B downloads, previews AI-generated subtitles by Hard_Space on Thursday January 09, 2025 @07:46AM (#42644762)
That doesn't work for idioms, certainly in Italian, which has multiple colorful metaphors which would be mystifying if translated directly.
Re:VLC tops 6B downloads, previews AI-generated subtitles by saint_yossarian on Thursday January 09, 2025 @08:10AM (#42644982)
I don't think anybody's talking about translations.
Re:VLC tops 6B downloads, previews AI-generated subtitles by entropie on Thursday January 09, 2025 @07:32AM (#42644634)
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read.

Questionable. It drives me crazy to have subtitles that are paraphrase in a way that changes the meaning of statements.

Re:VLC tops 6B downloads, previews AI-generated subtitles by raincole on Thursday January 09, 2025 @07:56AM (#42644850)
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read

I really hope people to stop doing that.

Re:VLC tops 6B downloads, previews AI-generated subtitles by latexr on Thursday January 09, 2025 @08:10AM (#42644976)
> There's an art to subtitling that goes beyond mere speech-to-text processing.

Agreed.

> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read.

Hard no. If it’s the same language, the text you read should match the text you listen to. Having those not match makes parsing confusing and slow.

> Sometimes you need to name a voice as unknown, to avoid spoilers.

Subtitles don’t usually mention who’s talking, because you can see that. Taking the source of a voice is uncommon and not something I expect these system to get right anyway.

Re:VLC tops 6B downloads, previews AI-generated subtitles by lidavidm on Thursday January 09, 2025 @07:45AM (#42644749)
I suppose this feature should have been termed closed captioning and not subtitling. It seems you're not going to get much sympathy for human translation here.
Re:VLC tops 6B downloads, previews AI-generated subtitles by lm28469 on Thursday January 09, 2025 @07:36AM (#42644677)
As long as they're synced properly I don't care much, some movies/shows have really bad sound mix and it's not always possible to find good subs in the first place
Re:VLC tops 6B downloads, previews AI-generated subtitles by mouse_ on Thursday January 09, 2025 @08:36AM (#42645289)
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read.

Pretty sure this is a violation of the Americans with Disabilities Act, so illegal in the U.S. at least. Being Deaf doesn't mean you need "reduced" dialogue.

Re:VLC tops 6B downloads, previews AI-generated subtitles by the_clarence on Thursday January 09, 2025 @08:40AM (#42645330)
The best subtitles I've had were fantrads
Re:VLC tops 6B downloads, previews AI-generated subtitles by latexr on Friday January 10, 2025 @06:08AM (#42654626)
Re:VLC tops 6B downloads, previews AI-generated subtitles by thiht on Thursday January 09, 2025 @09:02AM (#42645586)
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read

NO!

I speak and understand 90% of English but I still use subtitles because sometimes I don't understand a word, or the sound sucks, or the actor thought speaking in a very low voice was a good idea. When the subtitles don't match what's being said, it's a terrible experience.

Re:VLC tops 6B downloads, previews AI-generated subtitles by mohamez on Thursday January 09, 2025 @09:33AM (#42645958)
This is horrible for people who learn languages using TV Shows and Movies. One of the most frustrating things I've encountered while learning German is the "paraphrase" thing, it makes practicing listening very hard, because my purpose wasn't to understand what was being said, but rather familiarizing my ear with spoken German.

So, knowing exactly the words being said is of utter importance.