Isn't it just muscle memory? Like how it is difficult to switch back and forth between a Qwerty keyboard and a Dvorak keyboard, except with the addition of balance thrown in.

This is worth a watch, from Smarter Every Day.