I know what you mean. I use hjkl to navigate (vimium). Many sites who are careful about not overriding arrow keys don't imagine someone using other keys to navigate.