Same with pentadactyl. However with CTRL+z you can temporarily disable pentadactyl (I think it's the same shortcut in vimperator) until you press ESC.
If the developer is here and wants a workaround there is a way to make the game work with pentadactyl/vimperator without disabling them. The h/j/k/l keys do not work in insert mode, so on page load focus in a hidden textbox and capture keys pressed in the textbox.
Same here with Chrome's Vimium. Once the URL is excluded, it works fine.
Great idea though (once you turn off Vimperator) :) !