The few times that happens to me I just hit “i” to enter insert mode and try again.
I actually like using "i" to toggle enabling/disabling the extension, more convenient and powerful imo
You might already know, but if you click on the extension icon, you can specify which pages to disable it for. I disable it in pages that have useful keyboard navigations. But it's great for things like HN, where I can use J/K to navigate, then hit F and open a link I want.
I've used this in the past, and I've found that it's not a 1:1 replacement but rather an add-on to my browsing habits. It's great when having lunch with greasy fingers, I can do most things with one pinky press!
It's very easy to add exceptions for individual sites, for example letting youtube have f for youtube. You just click on the Vimium icon on toolbar and it pops up the dialog for that. Would take only about 5 seconds to do even while you are rowing.
Overall I don't want to come across as critical of of something that just needs a little effort to use, more of just setting expectations.
Normally this would be less of an issue but for me, I ran into this problem while on my rowing machine, and I didn't want to stop to fiddle with keybindings.