Re:Seamless: Meta's New Speech Models
by d3m0t3p on Friday December 01, 2023 @11:31AM
(#38488614)
Not necessarily true, for the first few sentences you won’t be able to do it. But afterwards, once the context is established you don’t really need to wait for the verb, you can predict it. For example if you are speaking about cleaning the house and you detail that you have cleaned the kitchen the stove and so on, you can predict the verb with only the start of the sentence. I don’t have any source to back this up, but it sounds plausible
[ Parent
| on HN ]
What if the predicted verb was incorrect, but the model has already translated the incorrect prediction? How does it tell you about a mistake?
A good approach might be to start with how top notch, ultra-experienced human translators handle corrections for real-time scenarios, for example, the expert translators that do the ear monitors at the United Nations. I've worked with a few such real-time translators when preparing keynote speeches and they seem to have rigorous processes that appeared quite deep. Probably a ton of domain expertise to be captured there.
That said, I suspect that real-time language translation is always going to be somewhat imperfect due to its nature. Non-real-time translation of literature is still a subjective art form even at the very high-end of human expertise.
Once you start predicting what someone is going to say you are no longer translating their speech
Yeah but then you're just introducing branch mispredictions which will cause latency and potential confusion down the line.
It's all a trade off.
Either way it's extremely exciting that we get to even discuss this stuff as real possiblities.