All sign languages use much more than handshape and position. And even outside of facial expression grammar (ie non-manual communication) there is a whole host of other complexities that make parsing sign language difficult.

Take classifiers: there are eight morphological types that allow users to express complex ideas without the use of any type of vocabulary-based utterances. You can certainly sign the equivalent of, “Did you see the car parked close to the tree,” (CAR PARK NEAR TREE SEE?) using vocabulary. Sure. It gets the point across, but it’s not efficient, and doesn’t really utilize sign language in a way that feels normal for users.

But that same question can be posed in a more condensed format if you use a classifier, (#CAR 5:CL-“tree”/3:CL-“car parked really close” SEE?). In the second utterance, one use the lexicalized (fingerspelled “car”), then holds up a five hand with bent elbow in their non-dominant hand, use the three hand representing the car placing it reeeeeeally close (with grammatical, NMC emphasis) to the tree, and finally asking the receiver if they saw it.

In that example alone, I’m using two types of classifiers: element and locative. Imagine a complex story using all eight classifier types with just a little vocabulary mixed in? I doubt AI/ML could interpret the context, characters, actions, events, descriptions, etc. accurately. Anyway, I hope this example shows there is a lot more complexity in computerized interpretation of signed languages well beyond gesture/vocabulary recognition.

Note: My glossing may not be perfect. It's been a while, and I’m a bit rusty.