I mean that it is not including features from the visible spectrum when making its decision, only IR & time of flight (depth). IR makes it easy to track the eyes (basically all eye tracking systems use IR) but misses skin tone cues.

A prop mask is one approach, but adversarial machine learning techniques might be even more effective, especially if you can study the adaption to a particular face.