From my experience, plugins add 'nice to haves', while the main benefit of vim is the core (modal editing, fast navigation, etc.).

If I get on a linux machine other than my workstation, I'll generally not install any plugins and still be able to get the job done without getting frustrated.