Agreed. I love vi, but this is almost always an anti-feature.

To disable, add these lines to .vimrc:

  map <C-a> <Nop>
  map <C-x> <Nop>