Developing with a text editor is definitely slower at first as you commit things to memory that an IDE may take care of, but in the long run I think it pays off since you're not bound to a particular tool. I've seen enough developers that can do next to nothing without an IDE. I don't want to be one of them.
So what you mention can indeed be an issue, but hardly a reason to dismiss IDEs in general.
I actually happen to use an IDE on a Java project every day at work. I think Java can be a difficult language to work with otherwise. It's verbose so you end up with an enormous amount of code and there's a very slow feedback cycle. These are areas where an IDE can really help.
I rarely find myself wanting an IDE when I'm coding in Perl, Python, Javascript or pretty much everything else I use. Then again, maybe it's because I haven't seen an IDE with much to offer other than code completion when working with these languages. I have almost zero need or want for code completion. For me, some of the best an IDE has to offer is the ability to find things quickly and refactoring. Many of them don't do a particularly good job at this with languages other than Java.
Even with less verbose languages, after a while there will be a fair amount of functions, objects (maybe), modules... Why would an IDE not be useful in navigating those?
I recently had a 500 lines Javascript program and some navigation features would have really helped (and 500 lines is not much).