Your comment had enough info that I was able to Google the answer: when you switch to vi mode, C-w gets switched to vi-unix-word-rubout. Override it in .inputrc.
Sorry, I wasn't clear enough. I re-setup the mappings after enabling vi mode. So I'm still mapping C-w to unix-word-rubout and not vi-unix-word-rubout.
Readline is great but I've been running into something that's driven me nuts for a few weeks now. I recently switched to editing-mode=vi and it seems to have changed the behavior of unix-word-rubout(normally ctrl-w). In emacs mode, unix-word-rubout seems to append to the kill ring. In vi mode, it only replaces it. So in emacs mode, I can eat several words with 'ctrl-w' and paste them with 'ctrl-y'. In vi mode(after setting up the mappings), 'ctrl-y' only pastes the last rubbed-out word.