Convert American sign language to Text
I am working on an app to help children in need to communicate with their caregivers. Most of these caregivers don't understand sign language.
I want to convert sign language to text and text to sign language. I know that this is an area of hot active research. I don't want to re-invent the wheel, I wanna use this as feature in my app where users enter input-text or an audio (using the speech to text api) then translate the text into a sign language corresponding to the text entered.
I am looking for an API, engine, open source or research project to embedded it, or a database in a form that could be easily translated to animation in 3D (DB of word to sign language 3D animation )
Can you please recommend any helpful tool/resource ?
In short, no I can't recommend, but I can suggest what some of the challenges are.
First of all, you have to know ASL has a way to represent english characters and numbers. This lets ASL speakers who know how to read English communicate with people who know how to fingerspell. But it's no more ASL then the alphabet is English.
ASL (and pidgins of English ADL such as SEE and PSE) is a language, and language translation is difficult all on it's own. ASL has some ways to code it to text, but it doesn't strictly have, AFAIK, a written form, which means there isn't a way to serialize it.
There are also ASL conventions that make it difficult to do translation. For example, in ASL I may establish a location, give it a description or a name, and then use it as a shorthand when communicating.
ASL also has a variety ways to communicate what is often communicated by tone or word emphasis. This can be in speed, size of movements, etc.
Also facial movements are important to ASL. They can be modifiers to signs, or signs all on their own. Also signs flow into one another... it's visual calligraphy. Beautiful, but it means that a translation system may need such transitions handled.
So when you read about people converting sign to text, it's generally some method of converting fingerspelling to letters with either a gesture capturing glove, or a vision system. I know there was a MIT project (http://dspace.mit.edu/bitstream/handle/1721.1/80082/43496321...) that did sign to text more in depth, and maybe you could reverse engineer it. I can't speak to text to sign, I know of no system other then websites with short gifs/video clips.
I'm sorry I can't be more helpful. I know there are some deaf hackers on here, you might see if they know about something, as I'm not hooked into the ASL/Deaf community.
I am not convinced, even if you have "good enough" text to sign representation of some sort, that you could generate movement. But good luck anyway.