Speaking as someone with a similar setup, I've never had many issues with plugins assuming the use of hjkl. The only one I can think of offhand that has issues is indent-object.

The thing that really causes issues is my attempt to swap the roles of [ and {, using [] to jump between paragraphs and {} to do fancy function jumping. Vim plugins really don't like that...