Compute vision is full of problems that turn out to be a lot harder than they look. Part of the reason is that our intuition about this is skewed by having access to a very good vision processing unit that we can't reproduce :)
Each finger has three joints that can bend in and a little bit back. Each finger can move from side to side. There is a small amount of rotation around the base of the finger. The thumb has three joints, the bottom of which is opposable to create a gripping hand. The entire hand can bend, yaw and rotate with respect to the arm, and there is further flexibility in the bones of the palm.
If you doubt how complicated the hand is, download a free 3D figure posing system like Daz Studio [1], instantiate a figure, and then try to make a hand grip a small object, or gesture (e.g. v-sign, 'ok', hitchhiking, vulcan greeting).
Even just ten years ago it was non-trivial to predict the shape of a room from a single 2d RGB image.
Human fingers only have a very narrow range of motion in a single direction. When the 2D profile of the hand changes, the set of finger movements to arrive at that profile is fairly predictable.