That sounds like an issue with specific plugins not loading on demand. Though it's not always avoidable, like when you open vim with specific files which triggers the lazy loading.