We had a student once who used a face detector to first find the face area, and then extracted the skin colour from that region for the subsequent hand tracking.