Actually there was a time when everbody knew the ASCII table and hjkl were the control codes for backspace, linefeed, vertical tab and formfeed which map pretty closely to the 4 arrow directions. The first two are keyboard oriented codes, the next two are more line-printer codes but the early teletype terminals were just a printer with a keyboard so it made perfct sense.

In the vi world, these keys didn't really mean anything. They were just easy mnemonic associations to memorize because there was a system of relatedness to all of it.