News for hackers. Stuff that matters.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by artninja1988 on Tuesday October 10, 2023 @12:44PM (#37834429)
I wonder what they use for the translation part
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by htrp on Tuesday October 10, 2023 @01:23PM (#37834864)
Assuming DeepL ..... not sure why they'd build NMT in house
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by leetharris on Tuesday October 10, 2023 @02:36PM (#37835744)
DeepL is very limited in context, only looking at a small surrounding sentence. Additionally, when you're trying to optimize for a more concise or verbose translation, you really need a generalist model.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by mcbuilder on Tuesday October 10, 2023 @03:17PM (#37836241)
The transformer model was invented to attend to context over the entire sequence length. Look at how the original authors used the Transformer for NMT in the original Vaswani et al publication. https://github.com/jadore801120/attention-is-all-you-need-py... See how the source sentence is run through the encoded, and then the decoder is run in an auto-regressive mode. It's been a while, so I can't recall if the encoded embeddings are masked to the same index to the decoder input sequence, and I don't see why you would even need to do particular task auto-regressively in theory. Found this survey, https://ar5iv.labs.arxiv.org/html/2204.09269 which outlines the pros and cons. A cool feature of transformers is their ability to pull in a great deal of surrounding context, look at how BERT was trained. Yeah, things are bit more boring with decoder only GPT types, but they are winning the day with the easy scaling.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by diminikolaou on Tuesday October 10, 2023 @02:39PM (#37835786)
Why not just use GPT-4?
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by bemmu on Tuesday October 10, 2023 @12:44PM (#37834430)
I gave it a try. Comment if you want me to try some other videos or target languages.

Elon Musk on Joe Rogan, speaking Finnish:

Original: https://www.youtube.com/watch?v=8Nael8xcSus

To Finnish: https://www.youtube.com/watch?v=OOkOafhAsDE

I'm a native Finnish speaker and while it starts off a bit strange, when they get to the mid to latter part it is really incredible, it's like an alternative universe where they were somehow born Finnish.

Paul Graham interview "What are some common mistakes founders make?"

Original: https://www.youtube.com/watch?v=845O4RE1XDM

To Japanese: https://www.youtube.com/watch?v=Rq_G0VXJ1xk

This one didn't work out so well. While some parts are decent, others sound like nonsense. But super cool it included pg blowing into the mic in the translation!

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by cyphar on Tuesday October 10, 2023 @01:04PM (#37834658)
Translating the Paul Graham video to Croatian works surprisingly well.

(As a non-native speaker) several bits of the Japanese transcription were just Japanese-sounding noise, and the prosody was completely all over the place in the bits that were actually Japanese. The dub does get across most of the things being talked about, but I wouldn't consider it an acceptable dub (even for most of the bits where it is actually in Japanese).

I wonder if reason is that the dubbing works better between more similar languages and struggles when dealing with disparate languages. I don't speak Chinese or Korean, but it would be interesting to see how well their dubs are.

Can you do another video into Japanese, to see if it's just an issue with the Paul Graham video? Also, how about translating from another language to English?

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by bemmu on Tuesday October 10, 2023 @02:08PM (#37835417)
Good idea, added a Putin video translated from Russian: https://www.youtube.com/watch?v=YSkjQJcqaFo
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by peddling-brink on Tuesday October 10, 2023 @05:49PM (#37838068)
I love the slow whispered "Thaaank Yooooouuuu" at :10.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by fivecour on Tuesday October 10, 2023 @05:19PM (#37837670)
Here's another example:

Steve Jobs on death | Walter Isaacson and Lex Fridman (EN->JP): https://www.youtube.com/watch?v=6YiJLDhcWSI

Maybe it does better on longer videos, but as someone who is fairly well-versed in Japanese this one was not a good result either. There is also a bit of wonkiness in this one, where in the dub Lex says some of Walter's lines and vice-versa, and at one point both of them repeat verbatim the same exact translated phrase which doesn't correspond to anything said in the original.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by d-lisp on Tuesday October 10, 2023 @01:54PM (#37835247)
Translating the Musk video to french is weird. What's interesting is that sometimes protagonists speak with an accent from Quebec, and sometimes with kind of an old school voice from some old french movie. There's a laugh in the sequence, which is translated with again some audio quality that belongs to an old french movie.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by bemmu on Tuesday October 10, 2023 @02:49PM (#37835921)
Back to the Future to Finnish: https://www.youtube.com/watch?v=fvvqUv6YYhc

I laughed because the voices are so out there, and then it made two little funny mistakes that caught me off guard when I was already laughing, so I could only switch to crying at that point.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by jjbinx007 on Tuesday October 10, 2023 @04:28PM (#37837040)
Here's Boris Johnson speaking German. It really does sound like his voice:

https://youtu.be/J4nxHqblzKA

I wonder if they're breaking Youtube's (and others) TOS by downloading the videos to work their AI magic on them?

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by tmikaeld on Tuesday October 10, 2023 @12:45PM (#37834434)
No matter what i try, even a new file from audacity gives "Watermark not allowed for audio input." error..
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by rexreed on Tuesday October 10, 2023 @02:15PM (#37835494)
I had the same problem. Then I watched the video and see that the upload is a video file not an audio file, so I uploaded as an MP4 and it worked. Seems like some sort of weird bug.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by DoingIsLearning on Tuesday October 10, 2023 @02:38PM (#37835771)
Doesn't seem like a bug at all.

They probably need to watermark the output video to clarify that the video is not original and is 'tampered' with.

The software is warning the user that it (rightly) refuses to watermark the video ... of an audio file.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by rexreed on Tuesday October 10, 2023 @02:41PM (#37835801)
The error message doesn't provide any sort of insight into whether or not that's the case. Contrary to that, the upload box says "Audio or Video File, up to 100MB". So that would imply that they aren't accepting only video input. While I agree with your suggestion that the reason why they seem to be accepting only video files is based on watermarking the output, the upload box seems to accept an audio file input. They should say "We only accept video file input".
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by lukasga on Tuesday October 10, 2023 @12:51PM (#37834517)
I saw that Spotify recently released a similar feature for selected podcasts, could anyone recommend some recent research papers related to voice translation? Would be really interested in reading!
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by lukasga on Tuesday October 10, 2023 @12:53PM (#37834532)
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by sdoering on Tuesday October 10, 2023 @12:55PM (#37834565)
I just tested it with a video of a Greek introductionary language lesson. Quite a lot of those examples actually did not work, as they seemed to have a problem with the words being spoken intentionally understandable, slower and in a way that while helpful to people not knowing the language, is not natural to the language.

But the things not working were actually sometimes really funny and the voice sounded more drunk than I would have imagined.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by el_isma on Tuesday October 10, 2023 @01:09PM (#37834703)
I've found their system tries to guess the language of the text, but there are many languages that use the same words, and so it will speak in the wrong language (or accent, even). I hope they make a solution for this, otherwise I don't think it will be suitable for production use cases.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by JonathanFly on Tuesday October 10, 2023 @04:23PM (#37836970)
It autodetects language by default but you can set to a specific one. Though you'd still have that problem on a multi-lingual input video.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by el_isma on Tuesday October 10, 2023 @05:45PM (#37838029)
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by hexage1814 on Tuesday October 10, 2023 @01:16PM (#37834776)
Okay, here are my thoughts: It sorta bad, but sometimes it is pretty goddammit good.

What bothered me the most is that it oftentimes when it transfers the "interpretation", it puts the intonation and the stress in the wrong part of sentence, it totally misses it. So it's just plain weird to see someone talking like that. But rarely, there are moments when it puts the word stress in the exactly right part of sentence and it looks like it was dubbed by a human. It's not many moments, but they are there, and you can have an idea of what this technology could look in its best version.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by abinaya_rl on Tuesday October 10, 2023 @01:41PM (#37835080)
I remember when this guy[1] launched TranslateAudio a few months ago. For those unfamiliar, it's the first tool that launched the voice translation. It was built by an Indiehacker.

[1] - https://twitter.com/mehulsharmamat

[2] - https://www.translateaudio.com/

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by JonathanFly on Tuesday October 10, 2023 @02:28PM (#37835655)
Genuinely impressive used as intended. Spectacular when not - English to English translation.

https://twitter.com/jonathanfly/status/1711607166371561805

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by qingcharles on Tuesday October 10, 2023 @03:28PM (#37836374)
That is hilarious, thank you.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by dsign on Tuesday October 10, 2023 @02:45PM (#37835874)
Is there any voice color/timbre change service that can be used for music production? Some sort of system that preserves the phonemes, the pitch and the timing of the input speech while changing the timbre.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by numpad0 on Tuesday October 10, 2023 @05:54PM (#37838138)
RVC WebUI / VC Client? Locally fine-tunable and runs at real time.

1: https://github.com/w-okada/voice-changer

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by mrweiner on Tuesday October 10, 2023 @08:34PM (#37839588)
Logic Pro has a plugin for this. Been years since I used it, but I think it was called a “formant shifter” or something to that effect
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by miramba on Tuesday October 10, 2023 @02:47PM (#37835902)
I guess I will live to see an iBabelfish…we will finally overcome jhwhs babylonian curse.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by Vespasian on Tuesday October 10, 2023 @02:54PM (#37835966)
Tried it on math video and it was surprisingly good at getting the speakers tone and the overall content correct.

Where it struggled a bit were terms like "X to the power of 7". Sometimes it chose the correct term and sometimes a more literal translation.

It had difficulties getting choosing a consistent translation of "you" (informal "du" or formal "Sie") and translated the "it's" variant incorrectly.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by cm2012 on Tuesday October 10, 2023 @02:58PM (#37836019)
This is like magic, wow.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by yieldcrv on Tuesday October 10, 2023 @03:04PM (#37836095)
nice! anybody thats still in the waiting list phase should get their money pulled by their VCs
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by grammers on Tuesday October 10, 2023 @03:05PM (#37836109)
This sounds extremely exciting. But on second thought I fear with this technology we'll be getting too lazy to actually learn a foreign language - which is also awesome.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by standardly on Tuesday October 10, 2023 @03:12PM (#37836178)
the vast majority of people are already too lazy to learn a second language... and those with the willpower to do so won't be stopped by this.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by fennecfoxy on Wednesday October 11, 2023 @11:07AM (#37845404)
I mean the Internet brought us all so much closer together than ever before; technology like this will make us even closer, if we can do better/perfect translations it means that we could be interacting with people in English who are native Spanish speakers, for example, without even realising (on the Internet, at least).

And it's tough, because if you like travelling, it's not really feasible to learn every single language for every place...but in the future if I can pop an earbud in & get a realtime translation & they can do the same, I'd probably speak to so many more strangers when I visit non English speaking countries. It'll be so great.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by coleman_ on Tuesday October 10, 2023 @03:24PM (#37836318)
I have been using dubbah which allows you to edit the translation as needed, and regenerate individual lines of dialogue
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by 6stringmerc on Tuesday October 10, 2023 @04:12PM (#37836860)
How does it handle translating copyrighted material? Disclaimer buried somewhere?
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by matthewbarras on Tuesday October 10, 2023 @04:50PM (#37837283)
What's the best "offline" TTS? 11Labs is great but I want to run locally
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by idonotknowwhy on Tuesday October 10, 2023 @05:08PM (#37837523)
I use silero for tts, and whisper (open source, local) for speech to text/translation.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by rexreed on Tuesday October 10, 2023 @05:38PM (#37837934)
Where are good examples of Silero TTS voices / outputs?
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by lordswork on Tuesday October 10, 2023 @05:03PM (#37837432)
Last I looked into it, tortoise-ttl
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by syntaxing on Tuesday October 10, 2023 @05:20PM (#37837689)
I want to convert articles and ebooks into audiobooks. What’s the best way to do this? Is there anything locally I can use?
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by ur-whale on Tuesday October 10, 2023 @05:37PM (#37837926)
One giant step forward towards peace on earth.

The day every Russian or Chinese can listen to English-speaking news in their native tongue will have a massive impact on how much tight a leash their local thuggish governments can keep on them.

Conversely, the day every American finally have access to non US-controlled world news automatically translated to English will be a giant step towards the US finally understanding how the rest of the world functions, and perhaps they'll finally become capable of deciding democratically to stop meddling in other people's affairs.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by cooper_ganglia on Tuesday October 10, 2023 @09:03PM (#37839778)
Perhaps I’m a bit too cynical about my own country, but I don’t see a day in the near future where the average American even cares about factually correct news instead of ideologically validating news, let alone going out of their way to translate foreign news to better understand the world; The people who care enough to do that are already doing that with existing technology. Making it more accessible probably won’t change too much, unfortunately.
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by fennecfoxy on Wednesday October 11, 2023 @11:13AM (#37845512)
Unfortunately, flag-carrying is hard-coded into our firmware. :/

It does make me laugh to think of some fun places we can stick USB cables to initiate a DFU.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by fennecfoxy on Wednesday October 11, 2023 @11:11AM (#37845485)
I'm not sure, it will definitely improve things, but insofar as the informational warfare part of the republican party in the US has shown us in the rest of the world, both Democrats and Republicans speak English and therefore the real barrier seems to be ideological rather than communication.

The only real solution is to remove our human/animal nature; a lot of people only care about something if it directly affects them. A lot of hot topic issues in the US at the moment seem to circle around controlling others (abortion, gay rights) and the freedom to do what you want even if it affects others (guns, owning an oversized vehicle).

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by gentleman11 on Tuesday October 10, 2023 @05:43PM (#37838004)
Their terms of service confused and creeped me out back when I looked into these people
Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by tkgally on Tuesday October 10, 2023 @08:11PM (#37839426)
I tried it on a video I made a few months ago. Here’s the original, in which I speak English:

https://www.youtube.com/watch?v=8JUepj7wIl0

And here are the first two and a half minutes with me dubbed into Japanese:

https://www.youtube.com/watch?v=z85FXwH6pUU

Aside from some pacing problems, the voice and intonation sound quite good to me. But anyone listening to the translation alone would find it hard to follow, as the tool seems to have mistranscribed some of my English and mistranslated some of the rest. (I understand Japanese.)

The dubbing would probably be a lot better if I had spoken English more slowly and with awareness that what I said would be interpreted into another language.

Re:ElevenLabs Launches Voice Translation Tool to Break Down ... by fennecfoxy on Wednesday October 11, 2023 @11:01AM (#37845328)
I think it did pretty well. No offense as your original video is still really great for a native English speaker (me) to listen to, but the English speech varies in pace a lot, sometimes you speak incredibly quickly, at such a pace that you even miss some syllables or blend them together for example https://youtu.be/8JUepj7wIl0?t=194 it sounds like you said "technogial" or "technodule" rather than "tech-no-log-ical". So I can understand that a machine may make mistakes on this; don't worry, as a Kiwi I mumble and do this too ha ha.

But the Japanese result is still already very very impressive even if I haven't spoken since I was 15 and over there on exchange (and stopped learning once I realise they romanise everything, why label a bottle with appuru jyusu when you have RINGO SHIRU!)