The indexing is what matters. After that, IntelliJ knows pretty much all about your project files. It only takes a minute on first load and reopening it will only scan what's changed which is fast.
I doubt IntelliJ plugins slow down the IDE noticeably, though I do keep the number of plugins activated to minimum for only what I use.
It's all nice and shiny until it takes several minutes to index your whole project structure. Then you can only hate it a little bit more every time you start it.
VSCode is not bloated (for now) with millions of options, I can add extensions and/or disable them whenever I need without losing performance or even creating my own ones without too much effort.