I switched to Colemak last year, and decided not to do any remapping. After a few weeks, I had no problems using Vim with the new layout. It helps that Colemak doesn't completely rearrange the keyboard.

Now my problem is that using Vim with any layout other than Colemak (read: QWERTY) is awkward.