Specifically, it was designed on the ADM3A keyboard, which had the arrows printed on the H, J, K and L keys. Also of note is the key in the upper right, which is labeled with "HOME", ~ and ^ (which explains why shells use "~/" to represent the users home directory).

And while I'm diving into history here, vi (which stands for "visual") was a modification of "ex" to use the newfangled "video terminal" instead of the battle tested teletype machine.