The indexing is what matters. After that, IntelliJ knows pretty much all about your project files. It only takes a minute on first load and reopening it will only scan what's changed which is fast.

I doubt IntelliJ plugins slow down the IDE noticeably, though I do keep the number of plugins activated to minimum for only what I use.