That may have something to do with it, but in practice, I find myself moving up/down (j/k) files far more often than I need to backtrack by a single character, and when I do, as often as not my pinky flies up to backspace (note this works by default only in Vim). It ends up being the most natural to keep your hands on the home row anyway.