Now that I notice that behavior it is really weird. Anyway, if that is a problem/annoyance for you, then you could try switching to insert-mode by using append ('a') instead, which will bump the cursor up a character and counter ESC bumping you down one.
It's not just Ctrl-[ that does this, Escape does it too. We just typically don't toggle between insert mode / ESC in vim fast enough and enough times to notice it. With the pedal it's quite easy to press and release it a bunch of times before you type anything.