Yeah, I think you really have to design your set of recognized gestures to make them have a low chance of ambiguity.
Hmmm, and yeah you could add templates that are variants of an intended output. To improve the chance of matching important gestures
It seems like it might be tricky to train it on what not to recognise. Or maybe you just need a lot of template gestures.