It may be partly the games fault, you are in a space ship with stuff around yourself that you need to physically look at to interact with (and by look at, I mean you need to point your head at it, there is no eye tracking), so i'll often take my hands off the keyboard, spin my head and use both hands on the joystick and buttons, then have to quickly find the home row again right after.

But I do think a part of it is that my brain has gotten very used to having the general location of the keyboard. So for instance I know the home row (for my left hand) is middle of the keyboard vertically, and on the left side horizontally. I'll quickly find the middle vertically by touch, but then I search for the nub on the f key next and if I was only a few keys too far to the right, I'll find j instead and then instantly assume I'm at the right spot, and it's quite jarring when it's wrong!

With sight, I'm able to know and move my hand toward the right hand side without feeling around more than a key in any direction for the most part.