News for hackers. Stuff that matters.
Re:Live-caption glasses let deaf people read conversations u... by jackblemming on Tuesday March 14, 2023 @04:03PM (#35157557)
Why can’t they also make glasses that translate sign language into English audio or text then?
Re:Live-caption glasses let deaf people read conversations u... by zie on Tuesday March 14, 2023 @04:12PM (#35157697)
Sign language is a complex language, so something has to learn yet another language. Perhaps made all the more complex as it's spatial and visual instead of verbal or written.

Not every deaf person uses sign language and there are many different sign languages in the world. American Sign Language(ASL) is but one of these languages.

Re:Live-caption glasses let deaf people read conversations u... by richbell on Tuesday March 14, 2023 @04:18PM (#35157792)
> Sign language is a complex language

Specifically, sign languages are not visual representations of existing languages (e.g. ASL and English) but completely different languages altogether.

Re:Live-caption glasses let deaf people read conversations u... by SamuelAdams on Tuesday March 14, 2023 @05:42PM (#35159262)
And even then, there are different dialects of ASL.
Re:Live-caption glasses let deaf people read conversations u... by zie on Tuesday March 14, 2023 @06:10PM (#35159712)
Exactly. Though Visual "representations"(for lack of a better word) of English do exist in dialects like SEE(Signed Exact English), which is obviously not the same as ASL.

I'm sure other signed languages have rough equivalents in their regional areas as well. I know Mexico has a few different signed languages, though I only have passing familiarity with 1 of their signed languages and it's definitely not a representation of Spanish.

Re:Live-caption glasses let deaf people read conversations u... by Daub on Tuesday March 14, 2023 @09:57PM (#35162707)
I was waiting for someone to say this. I have taught Humanities in a program which included a significant number of deaf students. Communication was enabled via translators. The difference in language culture plus the fact that everything was mediated through a third party made the whole experience profoundly unsatisfactory. Nuance was completely lost. Perhaps if I had recovered training in sign language things world have been different.
Re:Live-caption glasses let deaf people read conversations u... by dgunay on Tuesday March 14, 2023 @04:17PM (#35157780)
Not an expert in either technology but I would imagine a lot of the difficulties of speech to text (bad recording conditions, variety of accents & pronunciation differences, etc) also have analogues in computer recognition of hand signs (camera alignment is bad, cut off, lighting is bad, someone's hand signs are lazily performed or slightly different than textbook ASL, etc).

Speech to text was "solved" a long time ago but I've seen it take many years to become as usable as it has recently. And it still regularly is frustrating to use for me!

Re:Live-caption glasses let deaf people read conversations u... by tsimionescu on Tuesday March 14, 2023 @04:51PM (#35158358)
My experience of speech to text has been terrible. As soon as someone has a bit of an accent, it becomes completely incoherent, even if all participants understand the accent perfectly. Even for good accents, using any kind of technical terms or proper names throws it off. And even when none of these are a problem, it still has at least a few percent error, even for English.
Re:Live-caption glasses let deaf people read conversations u... by baseline-shift on Tuesday March 14, 2023 @08:28PM (#35161847)
true. But all transcription tools are terrible transcribing technical language if someone has accented English.
Re:Live-caption glasses let deaf people read conversations u... by dymk on Tuesday March 14, 2023 @04:12PM (#35157688)
I'm sure that's something someone could do! And it sounds like a very fun project. It probably hasn't already been done because there already exist very good speech-to-text models and not many (any?) sign-language-to-{text,audio} models
Re:Live-caption glasses let deaf people read conversations u... by jmcmaster on Tuesday March 14, 2023 @09:55PM (#35162698)
Microsoft Research had early implementation for this kind of visual sign language translation with Kinect & ASL recognition in 2013 or so. I expect that with the death of Kinect in the market it stayed lab-bound.
Re:Live-caption glasses let deaf people read conversations u... by quantumquetzal on Tuesday March 14, 2023 @10:29PM (#35162981)
Sign language (speaking with ASL in mind, unfamiliar with others) uses a different grammatical structure from American English. To boot, a signer can adopt different body languages to imbue their message with different tonality. Current ML models (as far as I’m aware) struggle with capturing that tonality in speech-to-text, and there’s _far_ less data for sign language recognition, regrettably.
Re:Live-caption glasses let deaf people read conversations u... by elil17 on Tuesday March 21, 2023 @04:57PM (#35252443)
Translating sign languages to spoken languages is really, really hard. Sign languages have a lot of features that don't really show up in spoken languages.

Let's take an example: the sign for "send email" in ASL (http://www.lifeprint.com/asl101/pages-signs/e/email.htm). If I point at you at the end of the sign it could mean "I will send you an email." If I point at myself it could mean "You should send me an email" or "Did you send me an email" depending on my facial expression. If I point off into space it could mean "I'm sending an email." If I start by pointing at you and then end the sign by pointing off into space it could mean "You should send an email." So your translator AI needs not only to understand the facial expressions and movements of the signer, but also the spacial relationships of everyone in the conversation. And that is just one aspect of the difficulty - there are many other features of sign languages that are just as hard to translate.

Perhaps this is the sort of thing that future AI systems could do. But it is quite complex.