You could use autocmd for filetypes that require plugins, so they don’t load for configs/texts. If your pm of choice allows this separation, ofc.

I’m not very experienced with complex plugins, but most of what I’ve seen did nothing unless explicitly initiated. Maybe there are global variables to control that.