So even if I only have to move x words to the left, delete until the first occurrence of character y, or change the text inside the very next quotation marks (one of my favorite tricks), the 'hassle' of switching between modes is worth the ease of moving around the document. In the few cases where I need to do something even more trivial, I can still navigate the 'regular' way in insert mode.
Furthermore, I find that I generally work better if I don't just rapidly and constantly switch (mental) modes, even for prose. I'm starting to feel my 'instant' connection to the output makes me sloppier and makes it harder for me to reason about what I'm exactly trying to do. So for me it even has a psychological benefit to use an editor that has modes.
For example in emacs you can move around, cut, paste, do minor manipulations like swapping words or case all while holding control down. When you want to enter text you let up.
Now I'm not a vim god, but I've used it enough that I have muscle memory that affects my use at the command line - I'll want to change something in a parenthesis and will type "ci(" (change inside parenthesis), only to find the literal characters appear... :)