[1] https://www.vlibras.gov.br/

I'm tying to make an analog to Google home like devices, but I can't imagine someone putting cameras all over a room to capture sign language from any angle.

Can someone help me understand what's the use case for computers understanding sign language that is not served by typing?

If you are born Deaf, you will not have the opportunity to natively acquire a spoken language. Our brains are not designed to acquire spoken language through writing -- and our spoken languages are designed to be acquired through listening, not reading.

It's easy to think that a Deaf signer living in America is a native speaker of English, but they very well may not be, having never heard the sounds and therefore been unable to internalize all of the many, many necessary grammatical and morphological rules of the language.

Expecting all Deaf people to simply use written English when dealing with a computer is like expecting all Americans to use Spanish to interact with their computers -- technically possible (Spanish is a perfectly good language, and relatively accessible starting even from grade school here), but ignores the human reality that people want (need) to speak their own language.

There are not yet standard ways to write sign languages. Partly this is because of difficulties in translating the medium to text -- unlike speech, sign language production is not entirely linear, in addition to the fact that your hands can be articulated comprehensibly in much more sophisticated ways than your mouth can.

Kind of a false equivalence here. For many Americans there is no real reason to learn Spanish. However, I have to imagine that for literally all deaf people learning to read and write the dominant local language is required in order to participate in the larger society. Signs will be written in that language, restaurant menus, newspapers, business emails, etc. You will be surrounded by writing of your local language every day.

There doesn't really exist a world where a person can get by without being able communicate with others in their vicinity. Deaf people are more often surrounded by non-deaf people who they will have to communicate with somehow. Sharing a common spoken language is kind of out of the question, sharing a common written language is the next best thing.

So, sure. Create a written sign language, make unicode support for it and create new locale settings for each dialect of sign language. This will not eliminate or even really reduce deaf people's need to learn to write their locally dominant language.

It really seems to me you are implying they can't read regular English - doesn't that make regular life quite a challenge? I would have assumed learning written English would be high priority.

Your answer just makes me even more confused. If they can't read English, what is the computer translating the sign language into? What are they doing with the results?

And this just brings up even more questions - how do they speak remotely to people? I was thinking that a video phone was an amazing advance, except you'd have to put the phone several feet away from you, making it pretty hard to see the screen.

Every single scenario I'm thinking of just makes the need for written language even more critical - otherwise such a person is almost entirely isolated, except when near other signing people.

This isn't even unique to the Deaf community. Something like 15 to 20 percent of the general population are considered functionally illiterate.

Loading the article directly into Firefox's reader mode by prepending about:reader?url= to the URL works fine though.

[0] https://twitter.com/archiveis/status/1018691421182791680

[1] https://support.mozilla.org/en-US/kb/firefox-dns-over-https

Is there something I'm missing? If it's such a big deal then I may work on it myself.

Sign language is at least three-dimensional, it can embed more information. The ASL signs for "forget" and "because" differ only in how you mouth them - easy enough, but sometimes you even skip mouthing them because there is sufficient context. What's worse is the location of the hands can vary depending on how lazy the signer is feeling, so "forget" is supposed to be signed from the forehead but it can also be signed from some empty point in front of your shoulder, off a "phantom person"'s imaginary forehead. You rely on the listener to understand that you intend for there to be a phantom person there.

Parsing sign language means you have to turn a 2D recording into a 3D model - even humans can have difficulty with this, parsing signs over Zoom - and you need to solve parsing facial expressions, reading lips and more. And you need the machine to be smart enough to detect the assignment of phantom persons to various locations around the signer. That's my impression as a deaf person anyway.

cough.

Take classifiers: there are eight morphological types that allow users to express complex ideas without the use of any type of vocabulary-based utterances. You can certainly sign the equivalent of, “Did you see the car parked close to the tree,” (CAR PARK NEAR TREE SEE?) using vocabulary. Sure. It gets the point across, but it’s not efficient, and doesn’t really utilize sign language in a way that feels normal for users.

But that same question can be posed in a more condensed format if you use a classifier, (#CAR 5:CL-“tree”/3:CL-“car parked really close” SEE?). In the second utterance, one use the lexicalized (fingerspelled “car”), then holds up a five hand with bent elbow in their non-dominant hand, use the three hand representing the car placing it reeeeeeally close (with grammatical, NMC emphasis) to the tree, and finally asking the receiver if they saw it.

In that example alone, I’m using two types of classifiers: element and locative. Imagine a complex story using all eight classifier types with just a little vocabulary mixed in? I doubt AI/ML could interpret the context, characters, actions, events, descriptions, etc. accurately. Anyway, I hope this example shows there is a lot more complexity in computerized interpretation of signed languages well beyond gesture/vocabulary recognition.

Note: My glossing may not be perfect. It's been a while, and I’m a bit rusty.

What would be a good source to get the full list of the eight morphological types, and how they are used?

https://www.allenisd.org/cms/lib/TX01001197/Centricity/Domai...

The linked PowerPoint does a good job of explaining a brief overview of the different types of classifiers. (I actually referenced it to correct my own answer because I knew something felt off. lol)

> Research led by a team at Microsoft, a big computing firm, and published in 2019, estimated that a typical publicly available corpus of a spoken language consists of around a billion words from as many as 1,000 different speakers. An equivalent data-set in a sign language might have fewer than 100,000 signs from just ten people. Besides large numbers, a good corpus also needs variety. This means conversations between native signers of diverse backgrounds, dialects and levels of fluency. Because deaf people more often have physical disabilities than do those with unaffected hearing, representing those with restricted fluency of movement is important.

An example sentence of mostly polysynthetic signs:

"i was out driving, just enjoying the scenery, the nature and all the trees passing by me," (3 signs) "then suddenly" (just raise the eyebrows and look scared at some point in front of you, with your hands still on the imaginary steering wheel), "my car crashed into a wall (or other immobile object) (and the hood got all deformed)" - 1 sign.

So that takes about 5 seconds to sign, but up to a minute to represent all the embedded information in English. But in practice, you'd skip a lot of the embedded information when translating to English, depending on what was actually important. So how do you make the machine know what was important? It depends on who you're translating for!