It worked out the opposite way for me. Using vim bindings keeps me in flow. If I don't have them I get aggravated, every time I have to slog through an inconvenient editing task because I changed my mind about some bit of code. It's not the seconds I save that makes me more productive, it's the mental state it helps me stay in.

It doesn't resonate with everybody, but I think it's going too far to say it's optimizing for the wrong thing.