I switched to Colemak last year, and decided not to do any remapping. After a few weeks, I had no problems using Vim with the new layout. It helps that Colemak doesn't completely rearrange the keyboard.
Now my problem is that using Vim with any layout other than Colemak (read: QWERTY) is awkward.
Interesting that you are also a Dvorak user. I originally learned to type on a Dvorak keyboard, and have only ever been able to hunt-and-peck on QWERTY. I imagine that vim's interface might seem like less of an incomprehensible, memory-dependent maze if one's fingers were accustomed to QWERTY. I tried to learn it once, with a little cheat sheet taped to my monitor, but I've never been any good at rote memorization, so I didn't really get anywhere with it and went back to nano.