After years of using Vim and Bash this was a relief that I learned way too late. For the lazy ones

  $ cat >~/.inputrc 
  set editing-mode vi
  set keymap vi

  $if mode=vi
    set keymap vi-insert
    "jj": vi-movement-mode
  $endif

Obviously you want to change vi-movement-mode to your favorite ESC style. E.g. searching through your last git commands becomes 'jj/git' then (you have to go to movement mode first, as insert mode is the default in the shell - which makes kind of sense).