It's kind of hard to explain, but I think my brain doesn't go "letter -> key on keyboard", it goes "letter -> location in keyboard layout -> key on keyboard".

If the key on keyboard stays the same, even though the layout changes, it really messes with the way my brain maps from letters to muscle movements, idk if this makes sense.